Oct 30 21:39:01.228753 ip-10-0-131-245 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:39:01.660252 ip-10-0-131-245 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:39:01.660252 ip-10-0-131-245 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:39:01.660252 ip-10-0-131-245 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:39:01.660252 ip-10-0-131-245 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664767    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664870    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664875    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664881    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664886    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664890    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664895    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664899    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664904    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664908    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664912    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664916    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664926    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664930    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664935    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664939    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664949    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664953    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664957    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:01.665441 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664962    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664966    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664970    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664974    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664978    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664988    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664992    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.664997    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665001    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665005    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665009    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665014    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665018    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665023    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665027    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665031    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665036    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665040    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665049    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665054    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:01.666005 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665058    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665062    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665067    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665071    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665075    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665079    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665083    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665088    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665092    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665096    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665101    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665110    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665115    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665119    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665123    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665128    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665133    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665138    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665142    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:01.666484 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665146    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665150    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665154    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665159    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665169    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665173    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665179    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665183    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665188    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665196    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665200    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665204    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665211    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665217    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665222    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665226    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665236    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665240    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665244    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:01.666927 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665249    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665253    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665257    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665261    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665265    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665270    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665274    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665278    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665283    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665287    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665296    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665300    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665304    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665308    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665312    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665316    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665320    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665327    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665333    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:01.667375 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665338    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665343    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665348    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665358    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665362    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665377    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665382    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665389    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665393    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.665402    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667449    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667460    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667483    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667487    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667490    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667493    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667496    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667499    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667501    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:01.667831 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667504    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667507    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667510    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667513    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667515    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667518    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667520    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667523    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667525    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667528    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667530    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667533    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667536    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667538    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667540    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667543    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667545    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667547    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667550    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667552    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:01.668284 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667555    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667559    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667573    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667577    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667581    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667584    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667587    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667589    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667592    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667594    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667597    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667599    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667601    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667604    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667607    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667610    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667613    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667616    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667619    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:01.668769 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667621    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667624    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667626    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667629    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667631    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667634    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667636    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667639    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667641    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667644    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667647    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667649    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667652    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667654    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667657    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667660    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667662    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667665    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667667    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667670    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:01.669219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667672    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667675    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667678    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667681    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667683    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667685    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667689    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667693    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667697    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667700    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667702    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667705    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667707    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667710    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667712    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667715    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667717    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667720    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667722    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667725    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:01.669721 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667727    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667730    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667732    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667735    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667737    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667740    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667742    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667745    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667748    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667750    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667752    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667755    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667757    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667761    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667763    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667766    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667768    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.667771    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668287    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:01.670185 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668292    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668296    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668299    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668302    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668304    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668307    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668309    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668312    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668315    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668317    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668320    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668322    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668325    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668327    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668329    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668332    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668334    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668337    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668340    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:01.670749 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668342    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668345    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668347    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668350    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668352    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668355    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668357    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668360    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668362    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668364    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668367    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668369    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668373    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668377    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668381    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668384    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668387    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668390    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668393    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668395    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:01.671217 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668398    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668400    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668403    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668405    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668408    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668410    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668413    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668415    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668417    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668420    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668423    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668425    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668428    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668430    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668433    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668435    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668438    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668440    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668442    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:01.671702 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668445    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668447    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668449    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668452    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668455    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668457    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668460    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668477    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668479    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668483    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668486    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668488    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668491    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668493    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668497    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668499    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668502    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668507    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668510    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668512    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:01.672134 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668515    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668517    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668520    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668522    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668525    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668528    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668530    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668533    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668536    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668540    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668542    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668545    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668548    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668550    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668553    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668555    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668559    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668562    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668565    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:01.672646 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668567    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668570    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668573    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668576    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668578    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668581    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668583    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.668586    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669636    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669645    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669652    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669657    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669662    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669665    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669669    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669673    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669676    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669679    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669683    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669686    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669689    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669691    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:39:01.673091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669694    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669697    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669700    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669703    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669708    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669711    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669714    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669717    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669721    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669725    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669728    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669731    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669734    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669737    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669745    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669748    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669751    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669754    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669759    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669762    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669765    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669768    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669771    2572 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669774    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669778    2572 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:39:01.673642 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669780    2572 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669783    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669786    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669789    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669797    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669800    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669803    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669806    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669809    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669812    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669814    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669817    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669820    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669823    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669826    2572 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669830    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669833    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669835    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669838    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669841    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669844    2572 flags.go:64] FLAG: --help="false"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669847    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669851    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669854    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:39:01.674269 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669857    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669860    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669863    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669866    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669869    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669871    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669874    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669877    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669880    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669883    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669885    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669888    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669891    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669893    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669896    2572 flags.go:64] FLAG: --lock-file=""
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669899    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669902    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669904    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669910    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669913    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669915    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669918    2572 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669921    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669925    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:39:01.674883 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669927    2572 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669930    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669934    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669937    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669941    2572 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669944    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669947    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669952    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669954    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669958    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669960    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669963    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669970    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669973    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669975    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669978    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669981    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669984    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669987    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669989    2572 flags.go:64] FLAG: --port="10250"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669992    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669995    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09c7820e47963d8d0"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.669998    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670001    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670004    2572 flags.go:64] FLAG: --register-node="true"
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670007    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:39:01.675447 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670011    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670014    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670016    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670019    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670023    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670026    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670029    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670032    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670035    2572 flags.go:64] FLAG: --runonce="false"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670038    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670041    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670044    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670048    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670051    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670055    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670058    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670061    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670064    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670067    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670070    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670073    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670076    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670079    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670081    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670086    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:39:01.676099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670089    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670092    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670113    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670117    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670120    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670123    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670126    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670129    2572 flags.go:64] FLAG: --v="2"
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670134    2572 flags.go:64] FLAG: --version="false"
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670138    2572 flags.go:64] FLAG: --vmodule=""
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670143    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670146    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670276    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670280    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670284    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670288    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670290    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670293    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670296    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670300    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670303    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670306    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:01.676707 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670312    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670316    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670320    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670324    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670326    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670329    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670332    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670334    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670337    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670339    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670342    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670344    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670347    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670350    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670352    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670355    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670358    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670360    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670363    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670365    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:01.677226 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670369    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670371    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670374    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670376    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670379    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670381    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670384    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670386    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670389    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670393    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670395    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670398    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670400    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670404    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670406    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670409    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670411    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670413    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670416    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670418    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:01.677741 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670421    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670423    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670426    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670428    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670430    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670433    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670435    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670438    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670440    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670442    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670445    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670447    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670452    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670454    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670457    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670459    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670462    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670481    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670485    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:01.678219 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670489    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670493    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670498    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670501    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670503    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670506    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670508    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670513    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670516    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670519    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670521    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670524    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670526    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670528    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670531    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670534    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670536    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670539    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670541    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:01.678668 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670544    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670546    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670549    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670551    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670553    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670556    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670560    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670562    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670565    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670567    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670569    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670572    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670574    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670577    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670579    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670582    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670585    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:01.679106 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670588    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.670614    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670652    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670656    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670659    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670662    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670666    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670668    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670671    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670673    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670676    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670678    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670681    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670683    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670685    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670688    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:01.679528 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670690    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670693    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670695    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670698    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670701    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670704    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670708    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670710    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670713    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670716    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670720    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670724    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670727    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670730    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670733    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670736    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670739    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670742    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670744    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670747    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:01.679921 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670749    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670752    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670755    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670758    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670760    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670763    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670765    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670768    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670770    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670773    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670775    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670778    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670780    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670783    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670785    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670788    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670790    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670794    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670796    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:01.680384 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670799    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670801    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670804    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670806    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670808    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670811    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670813    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670816    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670820    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670823    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670826    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670828    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670831    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670833    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670835    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670838    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670842    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670844    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670847    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:01.681758 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670851    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670854    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670857    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670860    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670862    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670865    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670867    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670870    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670872    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670875    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670877    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670881    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670883    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670886    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670889    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670891    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670894    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670896    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670898    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670901    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:01.682203 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670903    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670907    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670909    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670912    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670915    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670917    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670919    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670923    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670925    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670928    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670931    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670933    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670936    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.670938    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.678255    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.678269    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.678285    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.678290    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678344    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678349    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:01.682788 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678352    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678355    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678359    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678361    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678364    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678367    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678370    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678372    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678375    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678377    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678380    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678382    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678385    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678387    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678390    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678392    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678395    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678397    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678400    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678403    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:01.683251 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678405    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678413    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678416    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678418    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678421    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678424    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678426    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678430    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678432    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678435    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678438    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678440    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678443    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678445    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678448    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678451    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678453    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678456    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678458    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678461    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:01.683772 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678483    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678486    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678489    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678492    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678494    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678496    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678499    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678502    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678505    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678507    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678510    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678512    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678516    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678520    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678523    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678526    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678528    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678531    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678534    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:01.684230 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678537    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678540    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678542    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678545    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678547    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678550    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678552    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678555    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678558    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678560    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678563    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678565    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678567    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678570    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678574    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678578    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678581    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678584    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678586    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:01.684688 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678590    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678592    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678594    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678597    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678599    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678602    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678604    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678607    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678609    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678612    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678615    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678617    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678620    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678623    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678626    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678628    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678631    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678633    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678635    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:01.685125 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678638    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678640    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678643    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678646    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678648    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678651    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678653    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.678680    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678838    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678843    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678846    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678849    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678851    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678854    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678857    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678860    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:01.685578 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678863    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678865    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678868    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678870    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678873    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678875    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678878    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678882    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678886    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678889    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678893    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678895    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678898    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678900    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678903    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678906    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678908    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678911    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678913    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:01.685950 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678915    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678918    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678920    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678922    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678925    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678927    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678930    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678933    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678936    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678938    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678941    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678944    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678946    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678949    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678951    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678954    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678956    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678958    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678961    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678963    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:01.686405 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678965    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678968    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678971    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678973    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678976    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678979    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678981    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678983    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678986    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678988    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678990    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678993    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678995    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.678998    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679000    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679002    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679005    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679008    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679010    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679012    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:01.687033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679015    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679017    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679020    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679022    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679025    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679027    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679029    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679032    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679035    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679037    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679040    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679043    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679045    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679047    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679050    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679053    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679055    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679058    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679060    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679062    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:01.687505 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679065    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679067    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679069    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679072    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679074    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679077    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679080    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679082    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679084    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679087    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679089    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679092    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679094    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679097    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679099    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679102    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679106    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679109    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:01.688001 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679112    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.679134    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679167    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679170    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679173    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679175    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679178    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679180    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679183    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679186    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679188    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679191    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679193    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679196    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679198    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679200    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:01.688414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679203    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679206    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679209    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679212    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679214    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679216    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679218    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679221    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679223    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679226    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679229    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679231    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679234    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679236    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679238    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679241    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679243    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679246    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679248    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679250    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:01.688804 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679253    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679255    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679257    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679260    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679262    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679265    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679268    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679270    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679273    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679275    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679278    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679280    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679282    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679285    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679287    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679290    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679292    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679294    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679297    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679299    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:01.689256 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679302    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679305    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679307    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679310    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679312    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679315    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679317    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679319    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679322    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679324    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679326    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679329    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679331    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679333    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679336    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679338    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679341    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679343    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679345    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:01.689962 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679348    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679351    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679353    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679355    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679358    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679360    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679362    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679365    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679368    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679371    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679374    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679377    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679379    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679381    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679385    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679387    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679389    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679392    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679394    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:01.690752 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679397    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679400    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679402    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679404    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679407    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679409    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679412    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679414    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679419    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679422    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679424    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679427    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679429    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:01.679431    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.679497    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.681944    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.682955    2572 server.go:1014] "Starting client certificate rotation"
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.683057    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:39:01.691224 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.683085    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:39:01.691668 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.685410    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:39:01.692523 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.692496    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:39:01.707882 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.707861    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:39:01.711195 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.711181    2572 log.go:25] "Validated CRI v1 image API"
Oct 30 21:39:01.711718 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.711697    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:39:01.712083 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.712061    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:39:01.714809 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.714786    2572 fs.go:135] Filesystem UUIDs: map[2c53d666-2701-4b40-87d0-e258cf8f872c:/dev/nvme0n1p3 2f74cab7-b9c9-4b26-aab6-6562f111a755:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 21:39:01.714882 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.714807    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:39:01.719013 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.718895    2572 manager.go:217] Machine: {Timestamp:2026-10-30 21:39:01.718417241 +0000 UTC m=+0.377420134 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096400 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ffc6259e97cec50f46de6046a1ac1 SystemUUID:ec2ffc62-59e9-7cec-50f4-6de6046a1ac1 BootID:a557d771-d479-4fa4-90fa-1b76f99f7aae Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:55:0b:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:55:0b:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:9f:4f:8f:0d:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:39:01.719013 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.719004    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:39:01.719162 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.719146    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:39:01.722819 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.722798    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:39:01.722945 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.722821    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-245.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:39:01.722991 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.722947    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:39:01.722991 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.722956    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:39:01.722991 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.722979    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:39:01.723067 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.722997    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:39:01.724256 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.724246    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:39:01.728905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.728893    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:39:01.729017 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.729008    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:39:01.729054 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.729037    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:39:01.729054 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.729048    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:39:01.729115 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.729057    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:39:01.729115 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.729065    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:39:01.729115 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.729074    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:39:01.729370 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.729359    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:39:01.731838 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.731814    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:39:01.739861 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.739831    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:39:01.741204 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.740994    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:39:01.741264 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.741237    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:39:01.741455 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.741444    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:39:01.741520 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.741481    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:39:01.741520 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.741490    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:39:01.741520 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.741496    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:39:01.741520 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.741502    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:39:01.741520 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.741508    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:39:01.742166 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.742158    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:39:01.742196 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.742167    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:39:01.742196 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.742174    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:39:01.742196 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.742180    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:39:01.742196 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.742194    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:39:01.742317 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.742202    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:39:01.743107 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.743092    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:39:01.743107 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.743105    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:39:01.745351 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.745328    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cjbdk"
Oct 30 21:39:01.746297 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.746282    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-245.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:39:01.746611 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.746591    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-245.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:39:01.746722 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.746702    2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:39:01.746994 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.746981    2572 server.go:1263] "Started kubelet"
Oct 30 21:39:01.747097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.747053    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:39:01.747136 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.747068    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:39:01.747136 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.747126    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:39:01.747814 ip-10-0-131-245 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:39:01.748810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.748683    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:39:01.749150 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.749138    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:39:01.754056 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.754034    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.756076    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.756695    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.756807    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757510    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757523    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757521    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757656    2572 factory.go:55] Registering systemd factory
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757672    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757658    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757687    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757869    2572 factory.go:153] Registering CRI-O factory
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757890    2572 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757934    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757946    2572 factory.go:103] Registering Raw factory
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.757956    2572 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.759265    2572 manager.go:319] Starting recovery of all containers
Oct 30 21:39:01.762380 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.761382    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cjbdk"
Oct 30 21:39:01.766960 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.766928    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-245.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:39:01.768530 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.767191    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-245.us-east-2.compute.internal.18e36c78934fefd1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-245.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-245.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:39:01.746954193 +0000 UTC m=+0.405957066,LastTimestamp:2026-10-30 21:39:01.746954193 +0000 UTC m=+0.405957066,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-245.us-east-2.compute.internal,}"
Oct 30 21:39:01.773723 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.773706    2572 manager.go:324] Recovery completed
Oct 30 21:39:01.775966 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.775951    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:01.777460 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.777448    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:39:01.779067 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.779050    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:39:01.779134 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.779077    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:39:01.779134 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.779116    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:39:01.779697 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.779683    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:39:01.779697 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.779696    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:39:01.779813 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.779698    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:39:01.779813 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.779770    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:39:01.780942 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.780926    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:39:01.781011 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.780951    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:39:01.781011 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.780961    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:39:01.781939 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.781926    2572 policy_none.go:50] "Start"
Oct 30 21:39:01.781986 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.781948    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:39:01.781986 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.781963    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:39:01.784044 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.784032    2572 policy_none.go:44] "Start"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.826045    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.830787    2572 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.830812    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.830821    2572 server.go:87] "Starting device plugin registration server"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.831010    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.831021    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.831145    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.831280    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.831292    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.831721    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.831761    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.837561    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.839016    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.839066    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.839085    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:39:01.841844 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.839143    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:39:01.844083 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.843000    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:01.931377 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.931346    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:39:01.932178 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.932162    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:39:01.932251 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.932189    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:39:01.932251 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.932202    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:39:01.932251 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.932226    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.939914 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.939891    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"]
Oct 30 21:39:01.939998 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.939944    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:39:01.940939 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.940922    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:39:01.940993 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.940958    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:39:01.940993 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.940974    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:39:01.943017 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943001    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.943089 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.943020    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-245.us-east-2.compute.internal\": node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:01.943129 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943099    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:39:01.943285 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943272    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.943321 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943297    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:39:01.943881 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943852    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:39:01.943881 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943860    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:39:01.943881 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943884    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:39:01.944019 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943897    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:39:01.944019 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943884    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:39:01.944019 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.943964    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:39:01.945937 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.945921    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.946026 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.945945    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:39:01.946703 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.946683    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:39:01.946768 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.946707    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:39:01.946768 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.946717    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:39:01.958806 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.958778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.958806 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.958800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.958965 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:01.958818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1e9e5008f7a1caba4d99ed76ccdf2ff-config\") pod \"kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"a1e9e5008f7a1caba4d99ed76ccdf2ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.965532 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.965509    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:01.968836 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.968819    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found" node="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:01.970806 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:01.970790    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found" node="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.059419 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.059392    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.059584 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.059432    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.059584 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.059506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.059584 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.059535    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1e9e5008f7a1caba4d99ed76ccdf2ff-config\") pod \"kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"a1e9e5008f7a1caba4d99ed76ccdf2ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.059699 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.059589    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1e9e5008f7a1caba4d99ed76ccdf2ff-config\") pod \"kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"a1e9e5008f7a1caba4d99ed76ccdf2ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.059699 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.059595    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.065906 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.065886    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.166693 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.166644    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.267444 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.267424    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.273583 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.273568    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.276731 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.276718    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:02.367724 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.367697    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.468241 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.468168    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.568698 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.568682    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.669200 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.669173    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.683318 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.683294    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:39:02.683457 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.683437    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:39:02.683537 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.683462    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:39:02.770240 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.770176    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.770576 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.770209    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:34:02 +0000 UTC" deadline="2028-08-10 16:55:15.75897455 +0000 UTC"
Oct 30 21:39:02.770679 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.770578    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15595h16m12.988401819s"
Oct 30 21:39:02.870858 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:02.870810    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:02.895266 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:02.895236    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0788c9cb659ae56a4649dcd1f52d9f30.slice/crio-3f8a4fb8e510a96424c84147f74ea9866cc531496de47bfe3dbcaf8d09abfb87 WatchSource:0}: Error finding container 3f8a4fb8e510a96424c84147f74ea9866cc531496de47bfe3dbcaf8d09abfb87: Status 404 returned error can't find the container with id 3f8a4fb8e510a96424c84147f74ea9866cc531496de47bfe3dbcaf8d09abfb87
Oct 30 21:39:02.901368 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.901350    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:39:02.904659 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:02.904638    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e9e5008f7a1caba4d99ed76ccdf2ff.slice/crio-5ee4e37dee4e258080d4f0b20df83689a870fde8f925bfc4b036e1b70b6d678b WatchSource:0}: Error finding container 5ee4e37dee4e258080d4f0b20df83689a870fde8f925bfc4b036e1b70b6d678b: Status 404 returned error can't find the container with id 5ee4e37dee4e258080d4f0b20df83689a870fde8f925bfc4b036e1b70b6d678b
Oct 30 21:39:02.968083 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:02.968057    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:03.025504 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.025432    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:03.057429 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.057400    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:03.070766 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.070746    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:39:03.071848 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.071835    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:03.082186 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.082168    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:39:03.730617 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.730538    2572 apiserver.go:52] "Watching apiserver"
Oct 30 21:39:03.743948 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.743919    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:39:03.744278 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.744250    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-kqkkd","openshift-network-diagnostics/network-check-target-8k7kg","kube-system/global-pull-secret-syncer-r5xwr","kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s","openshift-dns/node-resolver-82lq5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-tgzlk","openshift-network-operator/iptables-alerter-2qlwc","openshift-ovn-kubernetes/ovnkube-node-kxgqs","openshift-image-registry/node-ca-kbfjc","openshift-multus/multus-additional-cni-plugins-zn8lv","openshift-multus/multus-wwgj9","kube-system/konnectivity-agent-sqkkh"]
Oct 30 21:39:03.746933 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.746911    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.749781 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.749461    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:03.749781 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.749539    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:03.751841 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.751754    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.751841 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.751809    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:03.753190 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.753172    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:39:03.753279 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.753245    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:39:03.753345 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.753315    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:39:03.753398 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.753367    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:39:03.753484 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.753464    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pnbm4\""
Oct 30 21:39:03.754097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.754074    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.756334 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.756313    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.757046 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.757028    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:39:03.759402 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.759067    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.761960 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.761934    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:03.764255 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.764237    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:03.764350 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.764291    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:03.767022 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.766997    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.769726 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.769701    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4989e2a1-ce7d-4da1-b626-946742bf4897-hosts-file\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.769839 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.769759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-netns\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.769839 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.769802    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-etc-kubernetes\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.769964 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.769855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:03.769964 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.769888    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-kubelet-config\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.769964 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.769909    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.769964 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.769935    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-tuned\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.770229 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.769975    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.770229 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770004    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-registration-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.770229 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770031    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4989e2a1-ce7d-4da1-b626-946742bf4897-tmp-dir\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.770229 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770076    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-multus-daemon-config\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.770229 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770105    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-modprobe-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.770229 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-kubernetes\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.770229 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770205    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-host\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770244    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-device-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770326    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-multus-certs\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysconfig\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770397    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswqn\" (UniqueName: \"kubernetes.io/projected/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kube-api-access-zswqn\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770422    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-system-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770464    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2plh5\" (UniqueName: \"kubernetes.io/projected/a11c4675-a114-40e1-a796-2443b92f1218-kube-api-access-2plh5\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.770608 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.770516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-tmp\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.771353 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.771314    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrb9\" (UniqueName: \"kubernetes.io/projected/4989e2a1-ce7d-4da1-b626-946742bf4897-kube-api-access-bnrb9\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.771472 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.771452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-bin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.771556 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.771523    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-multus\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.771605 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.771578    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-hostroot\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.771674 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.771659    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-conf-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.771723 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.771691    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-systemd\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.771783 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.771721    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-var-lib-kubelet\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.771783 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.771756    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.773304 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773280    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-socket-dir-parent\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.773413 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773352    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-kubelet\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.773413 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773379    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.773413 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773408    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-conf\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.773603 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-run\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.773603 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773475    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-sys\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.773603 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773503    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-cni-binary-copy\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.773603 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773527    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-k8s-cni-cncf-io\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.773603 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773586    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-lib-modules\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.773866 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773615    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-cnibin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.773866 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773644    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-dbus\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.773866 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vfg\" (UniqueName: \"kubernetes.io/projected/133bfc6c-cab2-4a9c-8710-e87f73f73804-kube-api-access-r9vfg\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.773866 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.773755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-socket-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.774074 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.774001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-sys-fs\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.774379 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.774351    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.776698 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.776642    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.777153 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.777130    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:39:03.796145 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.796105    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2vjg2"
Oct 30 21:39:03.808443 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.808393    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2vjg2"
Oct 30 21:39:03.845090 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.845047    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"a1e9e5008f7a1caba4d99ed76ccdf2ff","Type":"ContainerStarted","Data":"5ee4e37dee4e258080d4f0b20df83689a870fde8f925bfc4b036e1b70b6d678b"}
Oct 30 21:39:03.846420 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.846388    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"0788c9cb659ae56a4649dcd1f52d9f30","Type":"ContainerStarted","Data":"3f8a4fb8e510a96424c84147f74ea9866cc531496de47bfe3dbcaf8d09abfb87"}
Oct 30 21:39:03.858848 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.858825    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:39:03.862602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.862580    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:39:03.862945 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.862922    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:39:03.863178 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.863154    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:39:03.863296 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.863238    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:39:03.863296 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.863271    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:39:03.863409 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.863301    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:39:03.863409 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.863401    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:39:03.863693 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.863676    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:39:03.863833 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.863817    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:39:03.865409 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.865180    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:39:03.865409 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.865231    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:39:03.865409 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.865273    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:39:03.865409 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.865183    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:39:03.865706 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.865538    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:39:03.865706 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.865631    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:39:03.865706 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.865637    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:39:03.865863 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.865725    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:39:03.866049 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.866028    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n2fsk\""
Oct 30 21:39:03.867670 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.867648    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:39:03.867670 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.867656    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2pxp\""
Oct 30 21:39:03.868602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868414    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:39:03.868602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868427    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wpx2j\""
Oct 30 21:39:03.868602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868460    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:39:03.868602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868527    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jbnx4\""
Oct 30 21:39:03.868602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868550    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pp8j7\""
Oct 30 21:39:03.868602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868557    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:39:03.868602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868582    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:39:03.868602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868588    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9c426\""
Oct 30 21:39:03.869095 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868628    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:39:03.869095 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868881    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzrj6\""
Oct 30 21:39:03.869095 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.868975    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6cdkd\""
Oct 30 21:39:03.874584 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-env-overrides\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.874684 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874599    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswqn\" (UniqueName: \"kubernetes.io/projected/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kube-api-access-zswqn\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.874684 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874628    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-system-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.874684 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-binary-copy\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.874684 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874680    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-kubelet\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.874905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-systemd-units\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.874905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-tmp\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.874905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrb9\" (UniqueName: \"kubernetes.io/projected/4989e2a1-ce7d-4da1-b626-946742bf4897-kube-api-access-bnrb9\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.874905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874785    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-bin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.874905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874810    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-multus\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.874905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874838    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwl5t\" (UniqueName: \"kubernetes.io/projected/df49b636-0a33-4254-9692-3bc1870f181d-kube-api-access-xwl5t\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.874905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874866    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25874bd6-56b8-4935-80eb-c9fcb240ec56-iptables-alerter-script\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.874905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-bin\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874945    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.874974    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-script-lib\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875032    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-sys\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875059    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-system-cni-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875084    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25874bd6-56b8-4935-80eb-c9fcb240ec56-host-slash\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875124    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-systemd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875149    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-cnibin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875174    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12528140-5996-44d5-b1ea-a1eea5b4a65c-serviceca\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875199    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-sys-fs\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875230    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4989e2a1-ce7d-4da1-b626-946742bf4897-hosts-file\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-netns\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:03.875310 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875310    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-etc-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-konnectivity-ca\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875370    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-kubelet-config\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875394    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-tuned\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875420    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-registration-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875473    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-multus-daemon-config\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875499    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-agent-certs\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875524    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-netns\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875575    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-modprobe-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875610    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-host\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875636    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-device-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875660    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875697    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2plh5\" (UniqueName: \"kubernetes.io/projected/a11c4675-a114-40e1-a796-2443b92f1218-kube-api-access-2plh5\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875724    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-ovn\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875757    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttknd\" (UniqueName: \"kubernetes.io/projected/858066ef-e2da-42a9-9044-239570efa65c-kube-api-access-ttknd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875783    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysconfig\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.876097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875809    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cpd\" (UniqueName: \"kubernetes.io/projected/12528140-5996-44d5-b1ea-a1eea5b4a65c-kube-api-access-s7cpd\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875835    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-hostroot\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-conf-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qbm\" (UniqueName: \"kubernetes.io/projected/25874bd6-56b8-4935-80eb-c9fcb240ec56-kube-api-access-p6qbm\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875930    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-config\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-systemd\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.875984    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-var-lib-kubelet\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-socket-dir-parent\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876038    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-kubelet\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876063    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876105    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-conf\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876157    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-run\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876182    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-cni-binary-copy\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-k8s-cni-cncf-io\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876265    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-slash\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.876975 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876292    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-log-socket\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876320    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-lib-modules\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876347    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-var-lib-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-netd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876412    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12528140-5996-44d5-b1ea-a1eea5b4a65c-host\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876445    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-dbus\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876477    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vfg\" (UniqueName: \"kubernetes.io/projected/133bfc6c-cab2-4a9c-8710-e87f73f73804-kube-api-access-r9vfg\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876507    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-socket-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876533    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-etc-kubernetes\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876582    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-cnibin\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxcg\" (UniqueName: \"kubernetes.io/projected/98efb6cf-f6f5-4932-af9c-04640d6f338a-kube-api-access-jcxcg\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876644    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876672    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4989e2a1-ce7d-4da1-b626-946742bf4897-tmp-dir\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876702    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-node-log\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876733    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876762    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/858066ef-e2da-42a9-9044-239570efa65c-ovn-node-metrics-cert\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876792    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-kubernetes\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.877810 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876821    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-multus-certs\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.878632 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876851    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.878632 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.876879    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.878632 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.877334    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-system-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.878632 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.877716    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:39:03.878632 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.878080    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysconfig\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.878632 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.878127    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-hostroot\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.882699 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.882668    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-lib-modules\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.882947 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.882925    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-dbus\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.883375 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.883357    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-socket-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.883448 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.883410    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-etc-kubernetes\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.883519 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.883451    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-conf-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.883584 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.883566    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.884652 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.884624    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4989e2a1-ce7d-4da1-b626-946742bf4897-tmp-dir\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.884816 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.884772    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-tmp\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.884888 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.884834    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-kubernetes\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.884948 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.884902    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-multus-certs\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.884948 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.884896    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-netns\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.885054 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885034    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-bin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.885139 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885111    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-multus\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.885139 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885111    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.885287 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885209    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-sys\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.885287 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885248    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-cnibin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.885384 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885309    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-sys-fs\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.885384 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885348    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-var-lib-kubelet\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.885479 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885424    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-systemd\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.885547 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885512    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-modprobe-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.885596 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885555    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-host\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.885703 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885610    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-device-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.885703 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885680    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-registration-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.885802 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.885759    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-kubelet-config\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:03.886386 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.886362    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-multus-daemon-config\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.886491 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.886473    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-socket-dir-parent\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.886578 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.886531    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.887029 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.887005    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-cni-binary-copy\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.887127 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.887106    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:03.887215 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.887203    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret podName:aa1dbadd-b4a7-44cb-b35a-768e9315acfc nodeName:}" failed. No retries permitted until 2026-10-30 21:39:04.387175526 +0000 UTC m=+3.046178406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret") pod "global-pull-secret-syncer-r5xwr" (UID: "aa1dbadd-b4a7-44cb-b35a-768e9315acfc") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:03.887607 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.887531    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-conf\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.887678 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.887642    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-kubelet\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.887838 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.887816    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-k8s-cni-cncf-io\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.887904 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.887889    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4989e2a1-ce7d-4da1-b626-946742bf4897-hosts-file\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.888040 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.888021    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.888169 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.888153    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-run\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.889034 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.889013    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-tuned\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.898400 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.897196    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswqn\" (UniqueName: \"kubernetes.io/projected/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kube-api-access-zswqn\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:03.898400 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.897390    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:39:03.898400 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.897410    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:39:03.898400 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.897425    2572 projected.go:196] Error preparing data for projected volume kube-api-access-k4nf9 for pod openshift-network-diagnostics/network-check-target-8k7kg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:03.898400 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.897486    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9 podName:58d0b21d-3085-4cd7-9989-68c046410d00 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:04.397467024 +0000 UTC m=+3.056469904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k4nf9" (UniqueName: "kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9") pod "network-check-target-8k7kg" (UID: "58d0b21d-3085-4cd7-9989-68c046410d00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:03.898920 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.898897    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vfg\" (UniqueName: \"kubernetes.io/projected/133bfc6c-cab2-4a9c-8710-e87f73f73804-kube-api-access-r9vfg\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:03.900583 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.900562    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrb9\" (UniqueName: \"kubernetes.io/projected/4989e2a1-ce7d-4da1-b626-946742bf4897-kube-api-access-bnrb9\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:03.902884 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.902863    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2plh5\" (UniqueName: \"kubernetes.io/projected/a11c4675-a114-40e1-a796-2443b92f1218-kube-api-access-2plh5\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:03.977485 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-agent-certs\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:03.977666 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977522    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-netns\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.977666 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977560    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-ovn\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.977666 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977618    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-netns\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.977837 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977668    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttknd\" (UniqueName: \"kubernetes.io/projected/858066ef-e2da-42a9-9044-239570efa65c-kube-api-access-ttknd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.977837 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977702    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cpd\" (UniqueName: \"kubernetes.io/projected/12528140-5996-44d5-b1ea-a1eea5b4a65c-kube-api-access-s7cpd\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.977837 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977731    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qbm\" (UniqueName: \"kubernetes.io/projected/25874bd6-56b8-4935-80eb-c9fcb240ec56-kube-api-access-p6qbm\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.977837 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-config\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.977837 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977810    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978047 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977770    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-ovn\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978047 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977862    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.978047 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977925    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978047 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-slash\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978047 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.977973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-log-socket\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978047 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978002    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-var-lib-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978047 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978028    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-slash\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978047 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978028    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-netd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978065    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-log-socket\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978072    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-var-lib-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12528140-5996-44d5-b1ea-a1eea5b4a65c-host\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978116    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-netd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-cnibin\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978136    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12528140-5996-44d5-b1ea-a1eea5b4a65c-host\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978160    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxcg\" (UniqueName: \"kubernetes.io/projected/98efb6cf-f6f5-4932-af9c-04640d6f338a-kube-api-access-jcxcg\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978171    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-cnibin\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978189    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-node-log\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978215    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978251    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/858066ef-e2da-42a9-9044-239570efa65c-ovn-node-metrics-cert\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978281 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978281    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978303    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-node-log\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978308    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978351    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-env-overrides\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-binary-copy\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978415    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-kubelet\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978439    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-systemd-units\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwl5t\" (UniqueName: \"kubernetes.io/projected/df49b636-0a33-4254-9692-3bc1870f181d-kube-api-access-xwl5t\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25874bd6-56b8-4935-80eb-c9fcb240ec56-iptables-alerter-script\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978531    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-bin\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978558    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-config\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978569    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978579    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:03.978640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978622    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978653    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-script-lib\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-system-cni-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25874bd6-56b8-4935-80eb-c9fcb240ec56-host-slash\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-systemd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978764    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12528140-5996-44d5-b1ea-a1eea5b4a65c-serviceca\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-etc-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978818    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-kubelet\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-konnectivity-ca\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.978867    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-systemd-units\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.978933    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:03.979039 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:03.978986    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs podName:98efb6cf-f6f5-4932-af9c-04640d6f338a nodeName:}" failed. No retries permitted until 2026-10-30 21:39:04.478970198 +0000 UTC m=+3.137973076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs") pod "network-metrics-daemon-tgzlk" (UID: "98efb6cf-f6f5-4932-af9c-04640d6f338a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:03.979466 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.979381    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-bin\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.979676 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.979650    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-env-overrides\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.979821 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.979802    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-binary-copy\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.979884 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.979852    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.979884 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.979857    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25874bd6-56b8-4935-80eb-c9fcb240ec56-host-slash\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.979999 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.979915    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-system-cni-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.979999 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.979945    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.980209 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.980172    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25874bd6-56b8-4935-80eb-c9fcb240ec56-iptables-alerter-script\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.980209 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.980204    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-script-lib\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.980321 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.980248    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12528140-5996-44d5-b1ea-a1eea5b4a65c-serviceca\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:03.980321 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.980255    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-etc-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.980321 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.980284    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-systemd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.980502 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.980480    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.980869 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.980808    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-konnectivity-ca\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:03.980994 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.980945    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/858066ef-e2da-42a9-9044-239570efa65c-ovn-node-metrics-cert\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.983840 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.983794    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-agent-certs\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:03.993728 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.993703    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qbm\" (UniqueName: \"kubernetes.io/projected/25874bd6-56b8-4935-80eb-c9fcb240ec56-kube-api-access-p6qbm\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:03.996816 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.996796    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttknd\" (UniqueName: \"kubernetes.io/projected/858066ef-e2da-42a9-9044-239570efa65c-kube-api-access-ttknd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:03.997875 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.997844    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwl5t\" (UniqueName: \"kubernetes.io/projected/df49b636-0a33-4254-9692-3bc1870f181d-kube-api-access-xwl5t\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:03.997959 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.997898    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxcg\" (UniqueName: \"kubernetes.io/projected/98efb6cf-f6f5-4932-af9c-04640d6f338a-kube-api-access-jcxcg\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:03.999871 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:03.999845    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cpd\" (UniqueName: \"kubernetes.io/projected/12528140-5996-44d5-b1ea-a1eea5b4a65c-kube-api-access-s7cpd\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:04.063518 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.063463    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:04.076302 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.076268    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11c4675_a114_40e1_a796_2443b92f1218.slice/crio-7a46067642aa896d62e794c37362ecaf058f75d5192fac28d0023e33d52b912f WatchSource:0}: Error finding container 7a46067642aa896d62e794c37362ecaf058f75d5192fac28d0023e33d52b912f: Status 404 returned error can't find the container with id 7a46067642aa896d62e794c37362ecaf058f75d5192fac28d0023e33d52b912f
Oct 30 21:39:04.077445 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.077253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:04.087334 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.087305    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133bfc6c_cab2_4a9c_8710_e87f73f73804.slice/crio-2b9381caff5331d4a630ea5315b0b5575aabc300b7ae27e1a3f6568ad90907db WatchSource:0}: Error finding container 2b9381caff5331d4a630ea5315b0b5575aabc300b7ae27e1a3f6568ad90907db: Status 404 returned error can't find the container with id 2b9381caff5331d4a630ea5315b0b5575aabc300b7ae27e1a3f6568ad90907db
Oct 30 21:39:04.090400 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.090379    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:04.091498 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.091476    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:04.101156 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.100999    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:04.101856 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.101826    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b646a1_f444_48b7_b8ee_5cfd2b87c7a5.slice/crio-751088045a66d4c9a6cc10b2fba38fdfabf3772b7a75cbb9534a8252e24172e3 WatchSource:0}: Error finding container 751088045a66d4c9a6cc10b2fba38fdfabf3772b7a75cbb9534a8252e24172e3: Status 404 returned error can't find the container with id 751088045a66d4c9a6cc10b2fba38fdfabf3772b7a75cbb9534a8252e24172e3
Oct 30 21:39:04.105907 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.105885    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4989e2a1_ce7d_4da1_b626_946742bf4897.slice/crio-a8fd44ea15a811d8d286fe89c5d45834bf718467f289adf3284ee5b86616e873 WatchSource:0}: Error finding container a8fd44ea15a811d8d286fe89c5d45834bf718467f289adf3284ee5b86616e873: Status 404 returned error can't find the container with id a8fd44ea15a811d8d286fe89c5d45834bf718467f289adf3284ee5b86616e873
Oct 30 21:39:04.112314 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.112292    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:04.112913 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.112889    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4b5658_0527_4a3c_aa15_5f8d74b8d8a9.slice/crio-9509b71ca331cc8b14c716629e6976098a6eacd9cd0462ea0fcf5b51dbf3cc4b WatchSource:0}: Error finding container 9509b71ca331cc8b14c716629e6976098a6eacd9cd0462ea0fcf5b51dbf3cc4b: Status 404 returned error can't find the container with id 9509b71ca331cc8b14c716629e6976098a6eacd9cd0462ea0fcf5b51dbf3cc4b
Oct 30 21:39:04.123969 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.123940    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25874bd6_56b8_4935_80eb_c9fcb240ec56.slice/crio-2adcaad10cab6da1f15e0ea2cc246c12f639b06915ab24f740d673d973cdb738 WatchSource:0}: Error finding container 2adcaad10cab6da1f15e0ea2cc246c12f639b06915ab24f740d673d973cdb738: Status 404 returned error can't find the container with id 2adcaad10cab6da1f15e0ea2cc246c12f639b06915ab24f740d673d973cdb738
Oct 30 21:39:04.133182 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.133161    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:04.139905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.139887    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:04.144414 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.144393    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12528140_5996_44d5_b1ea_a1eea5b4a65c.slice/crio-2a04970af0aa5730957a35acfce921b0f7f44d0d27caac11eb31c63a64c0728d WatchSource:0}: Error finding container 2a04970af0aa5730957a35acfce921b0f7f44d0d27caac11eb31c63a64c0728d: Status 404 returned error can't find the container with id 2a04970af0aa5730957a35acfce921b0f7f44d0d27caac11eb31c63a64c0728d
Oct 30 21:39:04.146699 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.146679    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:04.156905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.156887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:04.157328 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.157300    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858066ef_e2da_42a9_9044_239570efa65c.slice/crio-a61dc660d3379e8175643c2080d20f402e47d2b277d50a5f78e07aaac93d9bb0 WatchSource:0}: Error finding container a61dc660d3379e8175643c2080d20f402e47d2b277d50a5f78e07aaac93d9bb0: Status 404 returned error can't find the container with id a61dc660d3379e8175643c2080d20f402e47d2b277d50a5f78e07aaac93d9bb0
Oct 30 21:39:04.168033 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:04.168000    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf49b636_0a33_4254_9692_3bc1870f181d.slice/crio-eacd06963ee33745f49385f0fdf6a99872dffdffa12c459c730dd9908ff2ff36 WatchSource:0}: Error finding container eacd06963ee33745f49385f0fdf6a99872dffdffa12c459c730dd9908ff2ff36: Status 404 returned error can't find the container with id eacd06963ee33745f49385f0fdf6a99872dffdffa12c459c730dd9908ff2ff36
Oct 30 21:39:04.259711 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.259634    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.483767    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.483839    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.483881    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:04.484064    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:04.484087    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:04.484101    2572 projected.go:196] Error preparing data for projected volume kube-api-access-k4nf9 for pod openshift-network-diagnostics/network-check-target-8k7kg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:04.484173    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9 podName:58d0b21d-3085-4cd7-9989-68c046410d00 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:05.484151434 +0000 UTC m=+4.143154299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4nf9" (UniqueName: "kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9") pod "network-check-target-8k7kg" (UID: "58d0b21d-3085-4cd7-9989-68c046410d00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:04.484648    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:04.484714    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret podName:aa1dbadd-b4a7-44cb-b35a-768e9315acfc nodeName:}" failed. No retries permitted until 2026-10-30 21:39:05.484694693 +0000 UTC m=+4.143697574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret") pod "global-pull-secret-syncer-r5xwr" (UID: "aa1dbadd-b4a7-44cb-b35a-768e9315acfc") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:04.484783    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:04.484865 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:04.484827    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs podName:98efb6cf-f6f5-4932-af9c-04640d6f338a nodeName:}" failed. No retries permitted until 2026-10-30 21:39:05.484810347 +0000 UTC m=+4.143813216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs") pod "network-metrics-daemon-tgzlk" (UID: "98efb6cf-f6f5-4932-af9c-04640d6f338a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:04.811706 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.811597    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:34:04 +0000 UTC" deadline="2028-06-18 05:25:23.301212855 +0000 UTC"
Oct 30 21:39:04.811706 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.811652    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14311h46m18.489564759s"
Oct 30 21:39:04.882661 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.882625    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"a61dc660d3379e8175643c2080d20f402e47d2b277d50a5f78e07aaac93d9bb0"}
Oct 30 21:39:04.887859 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.887825    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kbfjc" event={"ID":"12528140-5996-44d5-b1ea-a1eea5b4a65c","Type":"ContainerStarted","Data":"2a04970af0aa5730957a35acfce921b0f7f44d0d27caac11eb31c63a64c0728d"}
Oct 30 21:39:04.898115 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.898085    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qlwc" event={"ID":"25874bd6-56b8-4935-80eb-c9fcb240ec56","Type":"ContainerStarted","Data":"2adcaad10cab6da1f15e0ea2cc246c12f639b06915ab24f740d673d973cdb738"}
Oct 30 21:39:04.901507 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.901476    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kqkkd" event={"ID":"133bfc6c-cab2-4a9c-8710-e87f73f73804","Type":"ContainerStarted","Data":"2b9381caff5331d4a630ea5315b0b5575aabc300b7ae27e1a3f6568ad90907db"}
Oct 30 21:39:04.913273 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.913202    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wwgj9" event={"ID":"a11c4675-a114-40e1-a796-2443b92f1218","Type":"ContainerStarted","Data":"7a46067642aa896d62e794c37362ecaf058f75d5192fac28d0023e33d52b912f"}
Oct 30 21:39:04.917945 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.917918    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerStarted","Data":"eacd06963ee33745f49385f0fdf6a99872dffdffa12c459c730dd9908ff2ff36"}
Oct 30 21:39:04.920809 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.920783    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sqkkh" event={"ID":"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9","Type":"ContainerStarted","Data":"9509b71ca331cc8b14c716629e6976098a6eacd9cd0462ea0fcf5b51dbf3cc4b"}
Oct 30 21:39:04.932946 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.932917    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-82lq5" event={"ID":"4989e2a1-ce7d-4da1-b626-946742bf4897","Type":"ContainerStarted","Data":"a8fd44ea15a811d8d286fe89c5d45834bf718467f289adf3284ee5b86616e873"}
Oct 30 21:39:04.945466 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:04.945390    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s" event={"ID":"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5","Type":"ContainerStarted","Data":"751088045a66d4c9a6cc10b2fba38fdfabf3772b7a75cbb9534a8252e24172e3"}
Oct 30 21:39:05.494774 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.494736    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:05.494969 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.494810    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:05.494969 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.494845    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:05.495087 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.495004    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:39:05.495087 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.495033    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:39:05.495087 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.495048    2572 projected.go:196] Error preparing data for projected volume kube-api-access-k4nf9 for pod openshift-network-diagnostics/network-check-target-8k7kg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:05.495242 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.495108    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9 podName:58d0b21d-3085-4cd7-9989-68c046410d00 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:07.495090506 +0000 UTC m=+6.154093382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4nf9" (UniqueName: "kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9") pod "network-check-target-8k7kg" (UID: "58d0b21d-3085-4cd7-9989-68c046410d00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:05.495603 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.495541    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:05.495705 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.495612    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret podName:aa1dbadd-b4a7-44cb-b35a-768e9315acfc nodeName:}" failed. No retries permitted until 2026-10-30 21:39:07.49559648 +0000 UTC m=+6.154599344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret") pod "global-pull-secret-syncer-r5xwr" (UID: "aa1dbadd-b4a7-44cb-b35a-768e9315acfc") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:05.495705 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.495680    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:05.495827 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.495714    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs podName:98efb6cf-f6f5-4932-af9c-04640d6f338a nodeName:}" failed. No retries permitted until 2026-10-30 21:39:07.495701812 +0000 UTC m=+6.154704679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs") pod "network-metrics-daemon-tgzlk" (UID: "98efb6cf-f6f5-4932-af9c-04640d6f338a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:05.812789 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.812675    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:34:04 +0000 UTC" deadline="2028-07-29 06:20:23.554056016 +0000 UTC"
Oct 30 21:39:05.812789 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.812715    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15296h41m17.741343863s"
Oct 30 21:39:05.843398 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.842456    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:05.843398 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.842604    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:05.843398 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.843055    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:05.843398 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.843194    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:05.843398 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.843260    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:05.843398 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:05.843351    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:05.960865 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.960822    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"a1e9e5008f7a1caba4d99ed76ccdf2ff","Type":"ContainerStarted","Data":"2d5f4395e9e17c892f016d88891fea055b732d8ae7b1d2e9e8a29569be316ad2"}
Oct 30 21:39:05.976670 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.975859    2572 generic.go:356] "Generic (PLEG): container finished" podID="0788c9cb659ae56a4649dcd1f52d9f30" containerID="55850a9a26e900617508720e39c7f9d84fe54c6e77c008ee9767e4c5dff762b5" exitCode=0
Oct 30 21:39:05.976670 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:05.975913    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"0788c9cb659ae56a4649dcd1f52d9f30","Type":"ContainerDied","Data":"55850a9a26e900617508720e39c7f9d84fe54c6e77c008ee9767e4c5dff762b5"}
Oct 30 21:39:06.004578 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:06.002945    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal" podStartSLOduration=3.002924378 podStartE2EDuration="3.002924378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:05.982410738 +0000 UTC m=+4.641413633" watchObservedRunningTime="2026-10-30 21:39:06.002924378 +0000 UTC m=+4.661927272"
Oct 30 21:39:07.008321 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:07.008255    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"0788c9cb659ae56a4649dcd1f52d9f30","Type":"ContainerStarted","Data":"dcc7ef95a12557c54140644df06a870b1aafec0dab0fcce6619644993a9e8d6b"}
Oct 30 21:39:07.030064 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:07.030007    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal" podStartSLOduration=4.029985718 podStartE2EDuration="4.029985718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:07.029184523 +0000 UTC m=+5.688187415" watchObservedRunningTime="2026-10-30 21:39:07.029985718 +0000 UTC m=+5.688988604"
Oct 30 21:39:07.521264 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:07.521207    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:07.521426 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:07.521301    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:07.521426 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:07.521374    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:07.521570 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.521556    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:07.521653 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.521634    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs podName:98efb6cf-f6f5-4932-af9c-04640d6f338a nodeName:}" failed. No retries permitted until 2026-10-30 21:39:11.521613491 +0000 UTC m=+10.180616356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs") pod "network-metrics-daemon-tgzlk" (UID: "98efb6cf-f6f5-4932-af9c-04640d6f338a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:07.522167 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.522141    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:39:07.522167 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.522163    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:39:07.522315 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.522177    2572 projected.go:196] Error preparing data for projected volume kube-api-access-k4nf9 for pod openshift-network-diagnostics/network-check-target-8k7kg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:07.522315 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.522260    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9 podName:58d0b21d-3085-4cd7-9989-68c046410d00 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:11.522209933 +0000 UTC m=+10.181212810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4nf9" (UniqueName: "kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9") pod "network-check-target-8k7kg" (UID: "58d0b21d-3085-4cd7-9989-68c046410d00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:07.522430 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.522352    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:07.522430 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.522395    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret podName:aa1dbadd-b4a7-44cb-b35a-768e9315acfc nodeName:}" failed. No retries permitted until 2026-10-30 21:39:11.522381173 +0000 UTC m=+10.181384048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret") pod "global-pull-secret-syncer-r5xwr" (UID: "aa1dbadd-b4a7-44cb-b35a-768e9315acfc") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:07.856706 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:07.856671    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:07.856892 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.856870    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:07.857401 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:07.857357    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:07.857532 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.857490    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:07.857656 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:07.857609    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:07.857781 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:07.857747    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:08.018769 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:08.017227    2572 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="89d767db795f540365767a345abb4e98d21f93a293a0d179501be1dc1f0da3e1" exitCode=0
Oct 30 21:39:08.018769 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:08.018466    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerDied","Data":"89d767db795f540365767a345abb4e98d21f93a293a0d179501be1dc1f0da3e1"}
Oct 30 21:39:09.843650 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:09.842611    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:09.843650 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:09.842759    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:09.843650 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:09.843149    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:09.843650 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:09.843250    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:09.843650 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:09.843308    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:09.843650 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:09.843387    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:10.025397 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:10.024659    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kbfjc" event={"ID":"12528140-5996-44d5-b1ea-a1eea5b4a65c","Type":"ContainerStarted","Data":"1d4be2c355e1d357f6271af4db524ac56df4796be79d7cb2dc560acb089a1a0b"}
Oct 30 21:39:10.033775 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:10.033329    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sqkkh" event={"ID":"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9","Type":"ContainerStarted","Data":"ee026f200f64f316a4d4e987f7782cfcb485d4b04ca85a62289d7e626f823ba8"}
Oct 30 21:39:10.069099 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:10.068120    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kbfjc" podStartSLOduration=2.608689535 podStartE2EDuration="8.068098087s" totalImagesPullingTime="5.459408552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:10.045752166 +0000 UTC m=+8.704755053" watchObservedRunningTime="2026-10-30 21:39:10.068098087 +0000 UTC m=+8.727100974"
Oct 30 21:39:11.039636 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:11.039599    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s" event={"ID":"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5","Type":"ContainerStarted","Data":"38a101835c11292f2f44e3caa6728d5dcce85881c658500a6c62aa49b30b6ef4"}
Oct 30 21:39:11.561280 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:11.561237    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:11.561481 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:11.561311    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:11.561481 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:11.561368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:11.561641 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.561496    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:11.561641 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.561586    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs podName:98efb6cf-f6f5-4932-af9c-04640d6f338a nodeName:}" failed. No retries permitted until 2026-10-30 21:39:19.561562712 +0000 UTC m=+18.220565576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs") pod "network-metrics-daemon-tgzlk" (UID: "98efb6cf-f6f5-4932-af9c-04640d6f338a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:11.562116 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.562095    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:39:11.562203 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.562126    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:39:11.562203 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.562141    2572 projected.go:196] Error preparing data for projected volume kube-api-access-k4nf9 for pod openshift-network-diagnostics/network-check-target-8k7kg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:11.562203 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.562194    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9 podName:58d0b21d-3085-4cd7-9989-68c046410d00 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:19.562176532 +0000 UTC m=+18.221179396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4nf9" (UniqueName: "kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9") pod "network-check-target-8k7kg" (UID: "58d0b21d-3085-4cd7-9989-68c046410d00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:11.562375 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.562276    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:11.562375 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.562319    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret podName:aa1dbadd-b4a7-44cb-b35a-768e9315acfc nodeName:}" failed. No retries permitted until 2026-10-30 21:39:19.562304607 +0000 UTC m=+18.221307475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret") pod "global-pull-secret-syncer-r5xwr" (UID: "aa1dbadd-b4a7-44cb-b35a-768e9315acfc") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:11.842963 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:11.842893    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:11.843125 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.843053    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:11.843536 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:11.843511    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:11.843658 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.843637    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:11.844961 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:11.844933    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:11.845099 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:11.845073    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:12.345016 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:12.344969    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:39:12.548905 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:12.548824    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:12.549995 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:12.549972    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:12.568841 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:12.568782    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sqkkh" podStartSLOduration=5.834842592 podStartE2EDuration="10.56876481s" totalImagesPullingTime="4.733922218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:10.069056637 +0000 UTC m=+8.728059524" watchObservedRunningTime="2026-10-30 21:39:12.56876481 +0000 UTC m=+11.227767693"
Oct 30 21:39:12.837640 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:12.836629    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:39:12.344993096Z","UUID":"ef8aa37f-1bbc-4b3e-bb2b-bc1451a36d34","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:39:12.838399 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:12.838366    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:39:12.838918 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:12.838591    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:39:13.046596 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:13.046387    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wwgj9" event={"ID":"a11c4675-a114-40e1-a796-2443b92f1218","Type":"ContainerStarted","Data":"5deff041dfbec86a17c1dfc33829abd2da9477f6b8ff2efc86fab45592e14428"}
Oct 30 21:39:13.050393 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:13.050363    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerStarted","Data":"8b86f7a6bc1bf317c90e0908b6da5fa8cba36f95458beac93aaa119a6e63f83d"}
Oct 30 21:39:13.055163 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:13.055134    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-82lq5" event={"ID":"4989e2a1-ce7d-4da1-b626-946742bf4897","Type":"ContainerStarted","Data":"2a759ae609a8fbe19e002833cf100a3077144f2fa63ced481337b7f5926439f4"}
Oct 30 21:39:13.057188 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:13.057160    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s" event={"ID":"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5","Type":"ContainerStarted","Data":"5e546139d3e1649b96f0773ae9e72ed891f60e21503890e33aed099d6af227aa"}
Oct 30 21:39:13.097982 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:13.097666    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wwgj9" podStartSLOduration=2.975915445 podStartE2EDuration="11.097649689s" totalImagesPullingTime="8.121734244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:13.070131993 +0000 UTC m=+11.729134876" watchObservedRunningTime="2026-10-30 21:39:13.097649689 +0000 UTC m=+11.756652569"
Oct 30 21:39:13.839779 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:13.839695    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:13.840243 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:13.839815    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:13.840243 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:13.840211    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:13.840378 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:13.840305    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:13.840492 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:13.840453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:13.840591 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:13.840570    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:14.061579 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:14.061538    2572 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="8b86f7a6bc1bf317c90e0908b6da5fa8cba36f95458beac93aaa119a6e63f83d" exitCode=0
Oct 30 21:39:14.061795 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:14.061628    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerDied","Data":"8b86f7a6bc1bf317c90e0908b6da5fa8cba36f95458beac93aaa119a6e63f83d"}
Oct 30 21:39:14.064132 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:14.064096    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s" event={"ID":"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5","Type":"ContainerStarted","Data":"4280520d9466eb9b0fe4344ca43d27b33b1f1aaefb9e787d96d07483a3c4bb18"}
Oct 30 21:39:14.093297 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:14.092728    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-82lq5" podStartSLOduration=3.364586346 podStartE2EDuration="12.092714695s" totalImagesPullingTime="8.728128349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:13.117166265 +0000 UTC m=+11.776169148" watchObservedRunningTime="2026-10-30 21:39:14.092714695 +0000 UTC m=+12.751717578"
Oct 30 21:39:14.121780 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:14.121735    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s" podStartSLOduration=3.220566036 podStartE2EDuration="12.121717817s" totalImagesPullingTime="8.901151781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:14.120821548 +0000 UTC m=+12.779824431" watchObservedRunningTime="2026-10-30 21:39:14.121717817 +0000 UTC m=+12.780720700"
Oct 30 21:39:15.067959 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:15.067907    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2qlwc" event={"ID":"25874bd6-56b8-4935-80eb-c9fcb240ec56","Type":"ContainerStarted","Data":"c8c6e4df64d055844151579e53146b56dc910ec0eeafe01bc27f1c2e04b2beee"}
Oct 30 21:39:15.087524 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:15.087459    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2qlwc" podStartSLOduration=4.575854208 podStartE2EDuration="13.087446628s" totalImagesPullingTime="8.51159242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:15.087180748 +0000 UTC m=+13.746183633" watchObservedRunningTime="2026-10-30 21:39:15.087446628 +0000 UTC m=+13.746449510"
Oct 30 21:39:15.840379 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:15.840344    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:15.840564 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:15.840344    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:15.840564 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:15.840499    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:15.840564 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:15.840523    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:15.840722 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:15.840639    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:15.840762 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:15.840717    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:16.072903 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:16.072856    2572 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="d7245da925453a54b2431c77a9f8cb2f7909bc883395b2c8a3bacdb296b9f390" exitCode=0
Oct 30 21:39:16.073602 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:16.072937    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerDied","Data":"d7245da925453a54b2431c77a9f8cb2f7909bc883395b2c8a3bacdb296b9f390"}
Oct 30 21:39:16.074649 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:16.074605    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kqkkd" event={"ID":"133bfc6c-cab2-4a9c-8710-e87f73f73804","Type":"ContainerStarted","Data":"0ad6d4ace2c810d30e3be88e1a9ca2ef1ea9103191c720b7d9bcd25a0214bc66"}
Oct 30 21:39:16.126155 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:16.126056    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kqkkd" podStartSLOduration=2.924836907 podStartE2EDuration="14.126040945s" totalImagesPullingTime="11.201204038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:16.125548977 +0000 UTC m=+14.784551860" watchObservedRunningTime="2026-10-30 21:39:16.126040945 +0000 UTC m=+14.785043827"
Oct 30 21:39:17.079980 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:17.079901    2572 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="6491f7201c17c705f13d03eadbcaec745b60d6c3756b28fdcfb2323b79b374eb" exitCode=0
Oct 30 21:39:17.080567 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:17.080017    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerDied","Data":"6491f7201c17c705f13d03eadbcaec745b60d6c3756b28fdcfb2323b79b374eb"}
Oct 30 21:39:17.840429 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:17.840165    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:17.840663 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:17.840247    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:17.840663 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:17.840306    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:17.840663 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:17.840551    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:17.840798 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:17.840665    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:17.840798 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:17.840732    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:19.619404 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:19.619376    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:19.619410    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:19.619446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.619535    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.619551    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.619587    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.619607    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.619618    2572 projected.go:196] Error preparing data for projected volume kube-api-access-k4nf9 for pod openshift-network-diagnostics/network-check-target-8k7kg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.619594    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs podName:98efb6cf-f6f5-4932-af9c-04640d6f338a nodeName:}" failed. No retries permitted until 2026-10-30 21:39:35.619577283 +0000 UTC m=+34.278580157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs") pod "network-metrics-daemon-tgzlk" (UID: "98efb6cf-f6f5-4932-af9c-04640d6f338a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.619682    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret podName:aa1dbadd-b4a7-44cb-b35a-768e9315acfc nodeName:}" failed. No retries permitted until 2026-10-30 21:39:35.6196575 +0000 UTC m=+34.278660377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret") pod "global-pull-secret-syncer-r5xwr" (UID: "aa1dbadd-b4a7-44cb-b35a-768e9315acfc") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:39:19.619746 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.619703    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9 podName:58d0b21d-3085-4cd7-9989-68c046410d00 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:35.619693393 +0000 UTC m=+34.278696254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4nf9" (UniqueName: "kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9") pod "network-check-target-8k7kg" (UID: "58d0b21d-3085-4cd7-9989-68c046410d00") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:39:19.839695 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:19.839535    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:19.839778 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:19.839542    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:19.839832 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.839775    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:19.839881 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:19.839622    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:19.839916 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.839869    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:19.839964 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:19.839925    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:20.088851 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:20.088814    2572 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="18e5213e260edd8a41174f596197ed81a2b85b00260407b91f78e059cb7debdd" exitCode=0
Oct 30 21:39:20.088989 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:20.088896    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerDied","Data":"18e5213e260edd8a41174f596197ed81a2b85b00260407b91f78e059cb7debdd"}
Oct 30 21:39:20.095309 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:20.095278    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"c0ecf2c466f3a594fdc47e8c7e0cbb8b4a86a143b9dda384171589d9977f906e"}
Oct 30 21:39:20.095398 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:20.095369    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"91bc3a83a62b1e485d462adfb8caf85d115828cbdbaa42996241b7883a74d7a3"}
Oct 30 21:39:20.095435 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:20.095398    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"9af23011adbb1b2e291eaf104e05824c509c60bfe8b4bc6a2e629564f102c10a"}
Oct 30 21:39:20.095435 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:20.095425    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"83a190c6a84e41c58bd7e74a00d1e34a31cc9da4be88816bc93ae66755cb2a48"}
Oct 30 21:39:20.095520 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:20.095456    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"b99f95e624db721a8a0967e1b744f7c8e0c753d7273459115f4ec17dfe25a4cf"}
Oct 30 21:39:20.095520 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:20.095500    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"721643f9fa9bdf0e98809f355b9b5cdf0edd9156110777154e182957e6e2b86d"}
Oct 30 21:39:21.100414 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:21.100370    2572 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="ea34b14d32b90cdf2bf19068b06555bc18625bbe38286e86a1de6a120bbd4c04" exitCode=0
Oct 30 21:39:21.100895 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:21.100428    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerDied","Data":"ea34b14d32b90cdf2bf19068b06555bc18625bbe38286e86a1de6a120bbd4c04"}
Oct 30 21:39:21.840406 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:21.840370    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:21.840568 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:21.840521    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:21.840568 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:21.840534    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:21.840700 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:21.840639    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:21.840700 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:21.840649    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:21.840789 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:21.840727    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:22.109729 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:22.109656    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" event={"ID":"df49b636-0a33-4254-9692-3bc1870f181d","Type":"ContainerStarted","Data":"c5e0ac1501b07cf01da5b631bf7faeddbb601ee3affef9814b0e5faf92a62a83"}
Oct 30 21:39:22.112731 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:22.112709    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"60484ff411c6465a488139c2e572236e5205d320676300b7ebc39efdfbf0a460"}
Oct 30 21:39:22.140428 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:22.140380    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zn8lv" podStartSLOduration=8.089329841 podStartE2EDuration="20.140370449s" totalImagesPullingTime="12.051040608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:22.137722103 +0000 UTC m=+20.796724984" watchObservedRunningTime="2026-10-30 21:39:22.140370449 +0000 UTC m=+20.799373330"
Oct 30 21:39:22.911238 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:22.911203    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:22.911413 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:22.911348    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:39:22.911812 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:22.911793    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:23.839658 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:23.839627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:23.839658 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:23.839642    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:23.840138 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:23.839748    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:23.840138 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:23.839761    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:23.840138 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:23.839817    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:23.840138 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:23.839877    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:24.119957 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:24.119924    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" event={"ID":"858066ef-e2da-42a9-9044-239570efa65c","Type":"ContainerStarted","Data":"32acadba5b342282ab11128ffdef391c58e4ea67810735b28148542ed8ca6a34"}
Oct 30 21:39:24.120198 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:24.120175    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:24.120272 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:24.120236    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:24.135303 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:24.135281    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:24.159526 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:24.159482    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs" podStartSLOduration=6.710040628 podStartE2EDuration="22.159451099s" totalImagesPullingTime="15.449410471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:24.156708988 +0000 UTC m=+22.815711869" watchObservedRunningTime="2026-10-30 21:39:24.159451099 +0000 UTC m=+22.818453981"
Oct 30 21:39:25.122355 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.122186    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:25.138215 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.138190    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:25.139107 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.139085    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqkkh_cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9/konnectivity-agent/0.log" limit=1024
Oct 30 21:39:25.415628 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.415551    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqkkh_cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9/konnectivity-agent/0.log" limit=1024
Oct 30 21:39:25.843311 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.843282    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:25.843569 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.843356    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:25.843678 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:25.843597    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:25.843678 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:25.843633    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:25.843678 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.843362    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:25.843794 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:25.843702    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:25.897508 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.897459    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8k7kg"]
Oct 30 21:39:25.898091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.898068    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5xwr"]
Oct 30 21:39:25.899656 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:25.899635    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tgzlk"]
Oct 30 21:39:26.123463 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:26.123382    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:26.123463 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:26.123422    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:26.123963 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:26.123504    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:26.123963 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:26.123545    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:26.123963 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:26.123591    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:26.123963 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:26.123719    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:27.839837 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:27.839812    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:27.839837 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:27.839823    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:27.840236 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:27.839820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:27.840236 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:27.839905    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:27.840236 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:27.839998    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:27.840236 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:27.840059    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:28.199507 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:28.199452    2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sqkkh_cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9/konnectivity-agent/0.log" limit=1024
Oct 30 21:39:29.839627 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:29.839582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:29.840091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:29.839587    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:29.840091 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:29.839728    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:29.840091 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:29.839723    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:29.840091 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:29.839875    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:29.840091 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:29.840028    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:31.840234 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:31.840204    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:31.840707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:31.840273    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:31.840707 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:31.840306    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:31.840707 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:31.840345    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8k7kg" podUID="58d0b21d-3085-4cd7-9989-68c046410d00"
Oct 30 21:39:31.840707 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:31.840406    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgzlk" podUID="98efb6cf-f6f5-4932-af9c-04640d6f338a"
Oct 30 21:39:31.840707 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:31.840515    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r5xwr" podUID="aa1dbadd-b4a7-44cb-b35a-768e9315acfc"
Oct 30 21:39:33.186112 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.185933    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeReady"
Oct 30 21:39:33.186516 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.186195    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:39:33.839500 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.839450    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:33.839500 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.839489    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:33.839500 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.839500    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:33.851976 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.851951    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:39:33.852143 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.852040    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggbv4\""
Oct 30 21:39:33.852143 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.852037    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mh2p\""
Oct 30 21:39:33.852248 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.852230    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:39:33.852361 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.852325    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:39:33.852499 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.852394    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-scjcc\""
Oct 30 21:39:33.852499 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:33.852434    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:39:34.200615 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:34.200593    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-82lq5_4989e2a1-ce7d-4da1-b626-946742bf4897/dns-node-resolver/0.log"
Oct 30 21:39:35.199267 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:35.199245    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kbfjc_12528140-5996-44d5-b1ea-a1eea5b4a65c/node-ca/0.log"
Oct 30 21:39:35.644156 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:35.644128    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:35.644596 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:35.644161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:35.644596 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:35.644190    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:35.644596 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:35.644300    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:39:35.644596 ip-10-0-131-245 kubenswrapper[2572]: E1030 21:39:35.644380    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs podName:98efb6cf-f6f5-4932-af9c-04640d6f338a nodeName:}" failed. No retries permitted until 2026-10-30 21:40:07.644358949 +0000 UTC m=+66.303361828 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs") pod "network-metrics-daemon-tgzlk" (UID: "98efb6cf-f6f5-4932-af9c-04640d6f338a") : secret "metrics-daemon-secret" not found
Oct 30 21:39:35.647928 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:35.647900    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:35.648074 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:35.648057    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:35.708097 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:35.708077    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:35.715203 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:35.715183    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:36.064375 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:36.064346    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r5xwr"]
Oct 30 21:39:36.067208 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:36.067184    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8k7kg"]
Oct 30 21:39:36.067745 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:36.067722    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1dbadd_b4a7_44cb_b35a_768e9315acfc.slice/crio-c7090c90c084f844d964a13c4e062cc88bc94e913aa06554fa6a3e4650ec83f0 WatchSource:0}: Error finding container c7090c90c084f844d964a13c4e062cc88bc94e913aa06554fa6a3e4650ec83f0: Status 404 returned error can't find the container with id c7090c90c084f844d964a13c4e062cc88bc94e913aa06554fa6a3e4650ec83f0
Oct 30 21:39:36.069969 ip-10-0-131-245 kubenswrapper[2572]: W1030 21:39:36.069940    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d0b21d_3085_4cd7_9989_68c046410d00.slice/crio-8d79a6f7a118a594b4d35e48a838e9e3133b8846fd2e92d69172252c42829014 WatchSource:0}: Error finding container 8d79a6f7a118a594b4d35e48a838e9e3133b8846fd2e92d69172252c42829014: Status 404 returned error can't find the container with id 8d79a6f7a118a594b4d35e48a838e9e3133b8846fd2e92d69172252c42829014
Oct 30 21:39:36.147172 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:36.147140    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r5xwr" event={"ID":"aa1dbadd-b4a7-44cb-b35a-768e9315acfc","Type":"ContainerStarted","Data":"c7090c90c084f844d964a13c4e062cc88bc94e913aa06554fa6a3e4650ec83f0"}
Oct 30 21:39:36.148677 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:36.148654    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8k7kg" event={"ID":"58d0b21d-3085-4cd7-9989-68c046410d00","Type":"ContainerStarted","Data":"8d79a6f7a118a594b4d35e48a838e9e3133b8846fd2e92d69172252c42829014"}
Oct 30 21:39:40.162022 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:40.161979    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8k7kg" event={"ID":"58d0b21d-3085-4cd7-9989-68c046410d00","Type":"ContainerStarted","Data":"7da8bd0793f24dbad60f1331e37b403be5012d2cfe859ec7b159e416a591eb4c"}
Oct 30 21:39:40.162554 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:40.162131    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:40.181931 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:40.181881    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8k7kg" podStartSLOduration=34.646180359 podStartE2EDuration="38.181867827s" totalImagesPullingTime="3.535687468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:40.181647669 +0000 UTC m=+38.840650552" watchObservedRunningTime="2026-10-30 21:39:40.181867827 +0000 UTC m=+38.840870739"
Oct 30 21:39:41.168019 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:41.167932    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r5xwr" event={"ID":"aa1dbadd-b4a7-44cb-b35a-768e9315acfc","Type":"ContainerStarted","Data":"5a019d63d0e3fa622ba17f08f0fcd82f8309b2f7d24c3b545a016f13625d3bd2"}
Oct 30 21:39:41.188197 ip-10-0-131-245 kubenswrapper[2572]: I1030 21:39:41.188151    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r5xwr" podStartSLOduration=34.24982985 podStartE2EDuration="39.188138448s" totalImagesPullingTime="4.938308598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:41.187797657 +0000 UTC m=+39.846800540" watchObservedRunningTime="2026-10-30 21:39:41.188138448 +0000 UTC m=+39.847141330"
Oct 30 21:39:41.293987 ip-10-0-131-245 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:39:41.303093 ip-10-0-131-245 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:39:41.303261 ip-10-0-131-245 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:39:41.303307 ip-10-0-131-245 systemd[1]: kubelet.service: Consumed 2.088s CPU time.
Oct 30 21:39:41.312031 ip-10-0-131-245 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:39:41.383249 ip-10-0-131-245 kubenswrapper[4282]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:39:41.383249 ip-10-0-131-245 kubenswrapper[4282]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:39:41.383249 ip-10-0-131-245 kubenswrapper[4282]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:39:41.383249 ip-10-0-131-245 kubenswrapper[4282]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:39:41.385343 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385270    4282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:41.385343 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385345    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385349    4282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385352    4282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385355    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385358    4282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385360    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385363    4282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385373    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385376    4282 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385378    4282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385381    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385384    4282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385387    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385390    4282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385393    4282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385395    4282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385398    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385400    4282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385403    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:41.385414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385406    4282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385409    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385411    4282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385414    4282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385416    4282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385419    4282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385422    4282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385425    4282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385428    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385430    4282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385433    4282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385435    4282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385438    4282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385440    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385443    4282 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385446    4282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385448    4282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385450    4282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385453    4282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385456    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:41.385903 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385459    4282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385461    4282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385480    4282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385484    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385487    4282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385489    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385492    4282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385494    4282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385496    4282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385499    4282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385502    4282 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385505    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385507    4282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385510    4282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385514    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385517    4282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385519    4282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385522    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385526    4282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:41.386597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385530    4282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385533    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385536    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385539    4282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385542    4282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385544    4282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385547    4282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385549    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385552    4282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385554    4282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385557    4282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385560    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385562    4282 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385565    4282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385568    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385570    4282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385572    4282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385575    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385577    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:41.387074 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385580    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385583    4282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385586    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385589    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385591    4282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385594    4282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385596    4282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385599    4282 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385602    4282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385605    4282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385607    4282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385610    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385612    4282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385615    4282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385617    4282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385620    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385622    4282 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385625    4282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385627    4282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385630    4282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:41.387566 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385632    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385635    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385637    4282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385640    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385643    4282 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385646    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385649    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.385652    4282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386590    4282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386599    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386602    4282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386606    4282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386610    4282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386613    4282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386617    4282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386620    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386622    4282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386625    4282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386627    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386630    4282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:41.388047 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386633    4282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386635    4282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386638    4282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386641    4282 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386644    4282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386647    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386649    4282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386652    4282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386654    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386657    4282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386660    4282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386662    4282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386665    4282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386667    4282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386670    4282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386673    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386676    4282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386679    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386682    4282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386684    4282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:41.388564 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386688    4282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386691    4282 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386693    4282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386696    4282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386698    4282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386708    4282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386712    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386716    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386718    4282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386721    4282 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386724    4282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386726    4282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386729    4282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386732    4282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386735    4282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386737    4282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386740    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386742    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386745    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:41.389065 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386747    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386750    4282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386753    4282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386755    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386758    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386760    4282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386763    4282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386765    4282 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386768    4282 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386771    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386774    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386777    4282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386780    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386782    4282 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386790    4282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386793    4282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386796    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386799    4282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386801    4282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386803    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:41.389558 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386806    4282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386809    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386811    4282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386814    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386816    4282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386819    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386821    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386824    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386826    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386829    4282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386831    4282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386834    4282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386836    4282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386838    4282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386841    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386844    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386848    4282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386853    4282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386859    4282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:41.390049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386863    4282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386867    4282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386880    4282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386883    4282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386886    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386889    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386891    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386894    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386902    4282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386905    4282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386908    4282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386910    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386913    4282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386915    4282 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386921    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.386924    4282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387401    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387406    4282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387408    4282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:41.390514 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387411    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387414    4282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387416    4282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387419    4282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387422    4282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387424    4282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387428    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387431    4282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387433    4282 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387436    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387439    4282 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387442    4282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387445    4282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387448    4282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387452    4282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387455    4282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387459    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387479    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387485    4282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:41.390979 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387489    4282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387493    4282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387496    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387498    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387501    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387504    4282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387506    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387509    4282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387512    4282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387514    4282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387517    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387519    4282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387522    4282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387524    4282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387527    4282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387530    4282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387532    4282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387535    4282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387538    4282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:41.391449 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387540    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387543    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387546    4282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387548    4282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387551    4282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387554    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387556    4282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387559    4282 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387562    4282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387564    4282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387567    4282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387569    4282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387572    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387574    4282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387577    4282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387580    4282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387583    4282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387585    4282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387588    4282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387590    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:41.391984 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387593    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387595    4282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387598    4282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387600    4282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387603    4282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387605    4282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387608    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387612    4282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387615    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387618    4282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387622    4282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387624    4282 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387627    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387629    4282 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387632    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387635    4282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387637    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387640    4282 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387642    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387645    4282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:41.392486 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387647    4282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387650    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387652    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387654    4282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387657    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387660    4282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387665    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387667    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387670    4282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387673    4282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387675    4282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387678    4282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387680    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387683    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387686    4282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387688    4282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387690    4282 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387693    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387695    4282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387698    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:41.392968 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387700    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387704    4282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387707    4282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387709    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.387712    4282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387810    4282 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387817    4282 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387823    4282 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387828    4282 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387832    4282 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387835    4282 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387842    4282 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387846    4282 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387849    4282 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387852    4282 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387855    4282 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387859    4282 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387862    4282 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387867    4282 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387870    4282 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387873    4282 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387876    4282 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387879    4282 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:39:41.393456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387883    4282 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387886    4282 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387889    4282 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387892    4282 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387895    4282 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387899    4282 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387902    4282 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387905    4282 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387908    4282 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387912    4282 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387916    4282 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387919    4282 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387922    4282 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387925    4282 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387929    4282 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387931    4282 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387935    4282 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387937    4282 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387940    4282 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387943    4282 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387947    4282 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387950    4282 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387953    4282 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387956    4282 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:39:41.394193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387959    4282 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387963    4282 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387966    4282 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387971    4282 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387974    4282 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387977    4282 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387980    4282 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387983    4282 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387988    4282 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387991    4282 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387994    4282 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.387997    4282 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388001    4282 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388004    4282 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388008    4282 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388012    4282 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388015    4282 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388018    4282 flags.go:64] FLAG: --help="false"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388022    4282 flags.go:64] FLAG: --hostname-override="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388025    4282 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388028    4282 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388031    4282 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388034    4282 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:39:41.395137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388037    4282 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388040    4282 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388043    4282 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388046    4282 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388048    4282 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388052    4282 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388055    4282 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388057    4282 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388060    4282 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388063    4282 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388066    4282 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388069    4282 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388072    4282 flags.go:64] FLAG: --lock-file=""
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388075    4282 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388078    4282 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388080    4282 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388086    4282 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388089    4282 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388092    4282 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388095    4282 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388098    4282 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388101    4282 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388104    4282 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388106    4282 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388110    4282 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:39:41.395725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388113    4282 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388117    4282 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388122    4282 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388125    4282 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388128    4282 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388131    4282 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388133    4282 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388136    4282 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388139    4282 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388146    4282 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388149    4282 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388152    4282 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388155    4282 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388158    4282 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388160    4282 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388163    4282 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388166    4282 flags.go:64] FLAG: --port="10250"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388169    4282 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388172    4282 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09c7820e47963d8d0"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388175    4282 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388178    4282 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388180    4282 flags.go:64] FLAG: --register-node="true"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388183    4282 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388187    4282 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388190    4282 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388193    4282 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:39:41.396324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388196    4282 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388200    4282 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388203    4282 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388206    4282 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388209    4282 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388211    4282 flags.go:64] FLAG: --runonce="false"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388214    4282 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388217    4282 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388221    4282 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388224    4282 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388226    4282 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388229    4282 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388232    4282 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388235    4282 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388238    4282 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388241    4282 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388244    4282 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388247    4282 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388250    4282 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388252    4282 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388255    4282 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388260    4282 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388263    4282 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388265    4282 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388269    4282 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:39:41.396973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388271    4282 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388274    4282 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388277    4282 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388280    4282 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388282    4282 flags.go:64] FLAG: --v="2"
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388286    4282 flags.go:64] FLAG: --version="false"
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388290    4282 flags.go:64] FLAG: --vmodule=""
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388294    4282 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388298    4282 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388432    4282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388436    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388439    4282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388442    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388445    4282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388447    4282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388451    4282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388454    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388456    4282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388459    4282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388462    4282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388478    4282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:41.397623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388481    4282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388484    4282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388486    4282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388490    4282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388492    4282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388495    4282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388497    4282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388500    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388503    4282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388505    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388508    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388510    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388513    4282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388516    4282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388518    4282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388521    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388524    4282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388526    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388529    4282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:41.398122 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388532    4282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388535    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388537    4282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388540    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388543    4282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388545    4282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388548    4282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388552    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388555    4282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388557    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388560    4282 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388562    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388567    4282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388571    4282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388575    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388579    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388582    4282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388585    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388588    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:41.398597 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388591    4282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388593    4282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388596    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388598    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388601    4282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388604    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388606    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388609    4282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388611    4282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388614    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388616    4282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388619    4282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388622    4282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388624    4282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388627    4282 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388630    4282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388633    4282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388635    4282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388638    4282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388640    4282 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:41.399055 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388644    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388646    4282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388649    4282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388652    4282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388654    4282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388657    4282 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388659    4282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388662    4282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388666    4282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388669    4282 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388673    4282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388677    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388681    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388685    4282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388689    4282 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388704    4282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388708    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388710    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388713    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388715    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:41.399747 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388718    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388721    4282 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388723    4282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388726    4282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388728    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388731    4282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388733    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388735    4282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388740    4282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388744    4282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388746    4282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388749    4282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388752    4282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388754    4282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388757    4282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388759    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:41.400330 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.388786    4282 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388822    4282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388826    4282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388828    4282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388832    4282 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388835    4282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388838    4282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388840    4282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388843    4282 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388845    4282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388848    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388850    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388853    4282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388855    4282 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388858    4282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388861    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388863    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388866    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388868    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388870    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:41.401058 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388873    4282 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388875    4282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388878    4282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388881    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388883    4282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388886    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388889    4282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388893    4282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388896    4282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388899    4282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388902    4282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388904    4282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388907    4282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388909    4282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388912    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388914    4282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388918    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388921    4282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388923    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:41.401826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388926    4282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388929    4282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388931    4282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388934    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388936    4282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388939    4282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388941    4282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388944    4282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388946    4282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388949    4282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388951    4282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388954    4282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388956    4282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388959    4282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388961    4282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388964    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388967    4282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388969    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388972    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:41.402430 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388975    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388977    4282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388980    4282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388982    4282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388985    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388987    4282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388990    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388993    4282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388995    4282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.388998    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389002    4282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389005    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389007    4282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389010    4282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389012    4282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389016    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389019    4282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389022    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389025    4282 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:41.403188 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389028    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389032    4282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389035    4282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389038    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389041    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389043    4282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389046    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389049    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389051    4282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389054    4282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389056    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389059    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389062    4282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389064    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389067    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389070    4282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389073    4282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389076    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389078    4282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:41.403672 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389081    4282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389084    4282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389086    4282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389089    4282 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389092    4282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389095    4282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389097    4282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389100    4282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389102    4282 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389106    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.389108    4282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.393133    4282 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.393148    4282 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.393165    4282 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.393170    4282 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393247    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393252    4282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393255    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393258    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393261    4282 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:41.404171 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393264    4282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393266    4282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393269    4282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393272    4282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393274    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393277    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393281    4282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393290    4282 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393294    4282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393297    4282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393301    4282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393303    4282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393306    4282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393310    4282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393313    4282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393316    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393318    4282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393321    4282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393324    4282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:41.404678 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393326    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393329    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393331    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393334    4282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393337    4282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393340    4282 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393342    4282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393345    4282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393347    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393350    4282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393353    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393355    4282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393358    4282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393360    4282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393363    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393365    4282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393368    4282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393370    4282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393373    4282 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393375    4282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:41.405221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393377    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393386    4282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393388    4282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393391    4282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393393    4282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393396    4282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393398    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393402    4282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393405    4282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393408    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393411    4282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393413    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393416    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393419    4282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393421    4282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393424    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393427    4282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393429    4282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393432    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393434    4282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:41.405943 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393437    4282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393439    4282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393441    4282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393444    4282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393446    4282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393449    4282 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393451    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393453    4282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393456    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393459    4282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393462    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393483    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393487    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393490    4282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393502    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393506    4282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393508    4282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393511    4282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393514    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393516    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:41.406444 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393519    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393521    4282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393524    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393526    4282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393529    4282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393531    4282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393534    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393537    4282 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393539    4282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393542    4282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393545    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393547    4282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393550    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393552    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393555    4282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393557    4282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393560    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393562    4282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393565    4282 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:41.406956 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393567    4282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393570    4282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.393572    4282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.393600    4282 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394412    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394457    4282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394479    4282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394485    4282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394489    4282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394494    4282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394498    4282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394502    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394506    4282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394513    4282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394520    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:41.407415 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394532    4282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394536    4282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394540    4282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394545    4282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394549    4282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394553    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394558    4282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394562    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394566    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394570    4282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394574    4282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394579    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394583    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394592    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394597    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394601    4282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394605    4282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394609    4282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394614    4282 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:41.407796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394618    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394622    4282 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394626    4282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394657    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394661    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394665    4282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394674    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394678    4282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394683    4282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394687    4282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394691    4282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394695    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394699    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394703    4282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394708    4282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394712    4282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394716    4282 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394720    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394725    4282 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394733    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:41.408235 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394737    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394741    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394746    4282 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394750    4282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394755    4282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394759    4282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394763    4282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394767    4282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394771    4282 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394775    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394779    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394784    4282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394793    4282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394797    4282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394802    4282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394806    4282 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394847    4282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394852    4282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394856    4282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394860    4282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:41.408825 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394865    4282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394869    4282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394873    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394877    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394886    4282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394890    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394894    4282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394898    4282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394903    4282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394907    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394911    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394915    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394920    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394924    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394928    4282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394932    4282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394936    4282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394945    4282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394950    4282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:41.409313 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394955    4282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394959    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394963    4282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394970    4282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394977    4282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394982    4282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394987    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394991    4282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394995    4282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.394999    4282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395034    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395038    4282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395042    4282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395046    4282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395050    4282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395054    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:41.409775 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395059    4282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.395292    4282 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395355    4282 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395361    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395365    4282 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395369    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395372    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395376    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395379    4282 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395382    4282 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395384    4282 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395387    4282 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395390    4282 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395392    4282 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395395    4282 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395398    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:39:41.410162 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395401    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395404    4282 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395406    4282 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395409    4282 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395412    4282 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395415    4282 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395418    4282 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395421    4282 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395423    4282 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395426    4282 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395428    4282 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395432    4282 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395435    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395438    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395441    4282 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395443    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395446    4282 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395448    4282 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395451    4282 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395454    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:39:41.410681 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395457    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395459    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395484    4282 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395488    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395491    4282 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395494    4282 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395496    4282 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395499    4282 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395501    4282 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395506    4282 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395509    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395512    4282 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395516    4282 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395520    4282 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395524    4282 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395526    4282 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395529    4282 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395531    4282 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395534    4282 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:39:41.411446 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395536    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395539    4282 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395541    4282 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395544    4282 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395546    4282 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395549    4282 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395552    4282 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395555    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395557    4282 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395560    4282 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395562    4282 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395565    4282 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395567    4282 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395570    4282 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395572    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395575    4282 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395579    4282 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395582    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395585    4282 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:39:41.412072 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395588    4282 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395590    4282 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395593    4282 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395602    4282 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395605    4282 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395608    4282 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395610    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395613    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395615    4282 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395618    4282 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395620    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395623    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395626    4282 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395628    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395631    4282 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395633    4282 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395635    4282 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395639    4282 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395641    4282 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:39:41.412577 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395644    4282 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395646    4282 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395649    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395651    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395654    4282 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395656    4282 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395659    4282 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395663    4282 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395665    4282 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395668    4282 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395670    4282 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395673    4282 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395675    4282 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395678    4282 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:41.395680    4282 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.395737    4282 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.396746    4282 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.396797    4282 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.397008    4282 server.go:1014] "Starting client certificate rotation"
Oct 30 21:39:41.413049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.397166    4282 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:39:41.413532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.397212    4282 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:34:02 +0000 UTC" deadline="2028-05-07 00:37:20.953024725 +0000 UTC"
Oct 30 21:39:41.413532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.397247    4282 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13298h57m39.555779605s"
Oct 30 21:39:41.413532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.397481    4282 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:39:41.413532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.398515    4282 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:39:41.413532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.399973    4282 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:39:41.413532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.401748    4282 log.go:25] "Validated CRI v1 image API"
Oct 30 21:39:41.413532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.402551    4282 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:39:41.413532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.403976    4282 fs.go:135] Filesystem UUIDs: map[2c53d666-2701-4b40-87d0-e258cf8f872c:/dev/nvme0n1p3 2f74cab7-b9c9-4b26-aab6-6562f111a755:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.404000    4282 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a04970af0aa5730957a35acfce921b0f7f44d0d27caac11eb31c63a64c0728d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a04970af0aa5730957a35acfce921b0f7f44d0d27caac11eb31c63a64c0728d/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2adcaad10cab6da1f15e0ea2cc246c12f639b06915ab24f740d673d973cdb738/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2adcaad10cab6da1f15e0ea2cc246c12f639b06915ab24f740d673d973cdb738/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f8a4fb8e510a96424c84147f74ea9866cc531496de47bfe3dbcaf8d09abfb87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f8a4fb8e510a96424c84147f74ea9866cc531496de47bfe3dbcaf8d09abfb87/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ee4e37dee4e258080d4f0b20df83689a870fde8f925bfc4b036e1b70b6d678b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ee4e37dee4e258080d4f0b20df83689a870fde8f925bfc4b036e1b70b6d678b/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/751088045a66d4c9a6cc10b2fba38fdfabf3772b7a75cbb9534a8252e24172e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/751088045a66d4c9a6cc10b2fba38fdfabf3772b7a75cbb9534a8252e24172e3/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a46067642aa896d62e794c37362ecaf058f75d5192fac28d0023e33d52b912f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a46067642aa896d62e794c37362ecaf058f75d5192fac28d0023e33d52b912f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d79a6f7a118a594b4d35e48a838e9e3133b8846fd2e92d69172252c42829014/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d79a6f7a118a594b4d35e48a838e9e3133b8846fd2e92d69172252c42829014/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9509b71ca331cc8b14c716629e6976098a6eacd9cd0462ea0fcf5b51dbf3cc4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9509b71ca331cc8b14c716629e6976098a6eacd9cd0462ea0fcf5b51dbf3cc4b/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a61dc660d3379e8175643c2080d20f402e47d2b277d50a5f78e07aaac93d9bb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a61dc660d3379e8175643c2080d20f402e47d2b277d50a5f78e07aaac93d9bb0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8fd44ea15a811d8d286fe89c5d45834bf718467f289adf3284ee5b86616e873/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8fd44ea15a811d8d286fe89c5d45834bf718467f289adf3284ee5b86616e873/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7090c90c084f844d964a13c4e062cc88bc94e913aa06554fa6a3e4650ec83f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7090c90c084f844d964a13c4e062cc88bc94e913aa06554fa6a3e4650ec83f0/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eacd06963ee33745f49385f0fdf6a99872dffdffa12c459c730dd9908ff2ff36/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eacd06963ee33745f49385f0fdf6a99872dffdffa12c459c730dd9908ff2ff36/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12528140-5996-44d5-b1ea-a1eea5b4a65c/volumes/kubernetes.io~projected/kube-api-access-s7cpd:{mountpoint:/var/lib/kubelet/pods/12528140-5996-44d5-b1ea-a1eea5b4a65c/volumes/kubernetes.io~projected/kube-api-access-s7cpd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~projected/kube-api-access-r9vfg:{mountpoint:/var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~projected/kube-api-access-r9vfg major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25874bd6-56b8-4935-80eb-c9fcb240ec56/volumes/kubernetes.io~projected/kube-api-access-p6qbm:{mountpoint:/var/lib/kubelet/pods/25874bd6-56b8-4935-80eb-c9fcb240ec56/volumes/kubernetes.io~projected/kube-api-access-p6qbm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4989e2a1-ce7d-4da1-b626-946742bf4897/volumes/kubernetes.io~projected/kube-api-access-bnrb9:{mountpoint:/var/lib/kubelet/pods/4989e2a1-ce7d-4da1-b626-946742bf4897/volumes/kubernetes.io~projected/kube-api-access-bnrb9 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58d0b21d-3085-4cd7-9989-68c046410d00/volumes/kubernetes.io~projected/kube-api-access-k4nf9:{mountpoint:/var/lib/kubelet/pods/58d0b21d-3085-4cd7-9989-68c046410d00/volumes/kubernetes.io~projected/kube-api-access-k4nf9 major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volumes/kubernetes.io~projected/kube-api-access-ttknd:{mountpoint:/var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volumes/kubernetes.io~projected/kube-api-access-ttknd major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98efb6cf-f6f5-4932-af9c-04640d6f338a/volumes/kubernetes.io~projected/kube-api-access-jcxcg:{mountpoint:/var/lib/kubelet/pods/98efb6cf-f6f5-4932-af9c-04640d6f338a/volumes/kubernetes.io~projected/kube-api-access-jcxcg major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a11c4675-a114-40e1-a796-2443b92f1218/volumes/kubernetes.io~projected/kube-api-access-2plh5:{mountpoint:/var/lib/kubelet/pods/a11c4675-a114-40e1-a796-2443b92f1218/volumes/kubernetes.io~projected/kube-api-access-2plh5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa1dbadd-b4a7-44cb-b35a-768e9315acfc/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/aa1dbadd-b4a7-44cb-b35a-768e9315acfc/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5/volumes/kubernetes.io~projected/kube-api-access-zswqn:{mountpoint:/var/lib/kubelet/pods/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5/volumes/kubernetes.io~projected/kube-api-access-zswqn major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9/volumes/kubernetes.io~secret/agent-certs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df49b636-0a33-4254-9692-3bc1870f181d/volumes/kubernetes.io~projected/kube-api-access-xwl5t:{mountpoint:/var/lib/kubelet/pods/df49b636-0a33-4254-9692-3bc1870f181d/volumes/kubernetes.io~projected/kube-api-access-xwl5t major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b9b4f91a10622d06f56d5a2b19c6cdc36b4ff799d122ab4d1d64b2a2f3e9b8d8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/345776448fcbce4eb48d4fc777ba72240623bea10134d98ee648a005e6d5f239/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/9a1f70b6ac0115664b9187eba5d393b59afcfa19ead89948a3cecded8e984d45/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/15e6163f39d0ca4fb21a844fd72a4b8b6d3f72ea007adde3b92b74480bc2a715/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2d2a9279e9c05a4324eef7cac6ba10fd0346f4c99c3ab69d98cc289ec69c79a7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b06b513caf1c7d3042ab98673cafc4340981508925d36ab307411327c89f5d57/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/97d1ad47533b0a12ec039f6f89dae0e3374864e7afcda8c60abe734a1a31ef2a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c721e35e6ebe92b5db9ff3c085f67db64f9a49be469e80625bf9f4a973083768/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/24f76227104abb5a926077097668406c9a5fb61ea1a7d5c29d2520f117df9369/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e67edd95b07f61bfa4ec4f274889a083ec35f1d5c78a387f3032d8e4f6f89217/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/c4fa1fc78e420c6d08488f16446612e55aa5dc1a1effe7949826a09b7557a35f/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7d4f72010015e509a667cf98f192f396d183f326b173e1fca873799ddde630af/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a33749b2ce8f249cd26882cf2ec3e73f48b302a2f9b5618b116bad2996614a06/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/3d631b00c77d53e5941e7503686cc463642ea5c1b28f6d314b301fec1a743944/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/91312bb68feffb30834d5c35d154895792cab5a55bca68dde6df7e69d451d37e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e702d9e54e20813e615ea07680d384cb0216cb4ec4e80de6d42d0ef41a25e609/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/aec619aae9349bb8e4e18012060819e31c6d8d93e04e3d9fb757a67e7ba68a70/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6ceec34a167ca5548cf02f4c06a740766e43a003557a992191822577f741695a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/190518cde5db62e646124eb6fa68d9f44e72488d1193c1d052e21c1fbcacf126/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ce326e6ba473fe1b7c403e22898696bb0cf2772f026b5a58bc43b870ccde2564/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/02d42a5ecbb26fc89356c6d3e71efe96d6f8d5d0bed2e85254e2506880eb9b78/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/8b111f493565ad8a580a961a9657e370707522bbbb4587ac2c50829729a2451c/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/2869157c6c591f5ccebc923a70b588cfb70481dc6a6ddc2b5a2847b76a3602b0/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/96abdd57518a60d32b7b6444d26df648585fdd082c57d9135be1e5f6cf3aab40/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/892db1ffcfaac2749c6d8cecd250caaad971ec3d6f3d19f29eaa5dbbe5511e41/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/43c5d30cf9cb7fa31036b133e11e5d356cc205a1c2cba7ea98a4dc61e1aff9a4/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c0a6c10eb7f4196cd85db05ebc64b392a90f32123467bec0aabfc1e4f5a4eff5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/78ec2ccf909536846819eb3e58354d0f76dfd96c71e1942d1f1a92400938a752/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/d57a48ccf775d9b8c6220924d8b79c6bd0bced28f467a8d5fdb7c190798fa2c8/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/135923fd1e9563cbe882820f3fcadf6cc9ce530fe41937ae52dae5956c9a95bc/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/858549e5a907c6a1c400ff3a0ed2445bbb1be5ecf1412456bb883bb623295dc5/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0d7b423601b6d1baceef45345f81fdf2800358dca5a35a2f5355f85493cb0b80/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1c436c7da2e99594f6d49531a2d0506f24c592514f4bde154980c2a73d76bd82/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c781dc6af086daabd36d91593bd76cdf526ba511d4077269209dea1a264a0400/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d49b1201d9e13ca8585289630979596f32923a4899ee63e3e40881b5ba4fe08a/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.412279    4282 manager.go:217] Machine: {Timestamp:2026-10-30 21:39:41.41152408 +0000 UTC m=+0.069620300 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094770 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ffc6259e97cec50f46de6046a1ac1 SystemUUID:ec2ffc62-59e9-7cec-50f4-6de6046a1ac1 BootID:a557d771-d479-4fa4-90fa-1b76f99f7aae Filesystems:[{Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/751088045a66d4c9a6cc10b2fba38fdfabf3772b7a75cbb9534a8252e24172e3/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a61dc660d3379e8175643c2080d20f402e47d2b277d50a5f78e07aaac93d9bb0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58d0b21d-3085-4cd7-9989-68c046410d00/volumes/kubernetes.io~projected/kube-api-access-k4nf9 DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5/volumes/kubernetes.io~projected/kube-api-access-zswqn DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a11c4675-a114-40e1-a796-2443b92f1218/volumes/kubernetes.io~projected/kube-api-access-2plh5 DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a04970af0aa5730957a35acfce921b0f7f44d0d27caac11eb31c63a64c0728d/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa1dbadd-b4a7-44cb-b35a-768e9315acfc/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/98efb6cf-f6f5-4932-af9c-04640d6f338a/volumes/kubernetes.io~projected/kube-api-access-jcxcg DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9509b71ca331cc8b14c716629e6976098a6eacd9cd0462ea0fcf5b51dbf3cc4b/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~projected/kube-api-access-r9vfg DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4989e2a1-ce7d-4da1-b626-946742bf4897/volumes/kubernetes.io~projected/kube-api-access-bnrb9 DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ee4e37dee4e258080d4f0b20df83689a870fde8f925bfc4b036e1b70b6d678b/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/133bfc6c-cab2-4a9c-8710-e87f73f73804/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/12528140-5996-44d5-b1ea-a1eea5b4a65c/volumes/kubernetes.io~projected/kube-api-access-s7cpd DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8fd44ea15a811d8d286fe89c5d45834bf718467f289adf3284ee5b86616e873/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/25874bd6-56b8-4935-80eb-c9fcb240ec56/volumes/kubernetes.io~projected/kube-api-access-p6qbm DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eacd06963ee33745f49385f0fdf6a99872dffdffa12c459c730dd9908ff2ff36/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f8a4fb8e510a96424c84147f74ea9866cc531496de47bfe3dbcaf8d09abfb87/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d79a6f7a118a594b4d35e48a838e9e3133b8846fd2e92d69172252c42829014/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df49b636-0a33-4254-9692-3bc1870f181d/volumes/kubernetes.io~projected/kube-api-access-xwl5t DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7090c90c084f844d964a13c4e062cc88bc94e913aa06554fa6a3e4650ec83f0/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2adcaad10cab6da1f15e0ea2cc246c12f639b06915ab24f740d673d973cdb738/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a46067642aa896d62e794c37362ecaf058f75d5192fac28d0023e33d52b912f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/858066ef-e2da-42a9-9044-239570efa65c/volumes/kubernetes.io~projected/kube-api-access-ttknd DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:8d79a6f7a118a59 MacAddress:fe:f0:cc:f5:ae:76 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fb:55:0b:ef Speed:0 Mtu:9001} {Name:br-int MacAddress:92:ae:46:48:1a:eb Speed:0 Mtu:8901} {Name:c7090c90c084f84 MacAddress:fe:e8:30:ff:f8:be Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fb:55:0b:ef Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:76:fe:29:57:3e:5c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:9f:4f:8f:0d:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.412547    4282 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.412592    4282 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.412865    4282 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.412891    4282 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-245.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413005    4282 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413013    4282 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413023    4282 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413042    4282 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413122    4282 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413272    4282 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413381    4282 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413415    4282 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:39:41.413788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413426    4282 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:39:41.414359 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413435    4282 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:39:41.414359 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413444    4282 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:39:41.414359 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.413452    4282 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:39:41.414359 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.414153    4282 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:39:41.414359 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.414166    4282 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:39:41.414708 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.414677    4282 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:39:41.416410 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416397    4282 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:39:41.416410 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416410    4282 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:39:41.416526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416517    4282 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:39:41.416714 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416703    4282 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:39:41.416744 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416733    4282 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:39:41.416823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416815    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:39:41.416860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416829    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:39:41.416860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416835    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:39:41.416860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416841    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:39:41.416860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416847    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:39:41.416860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416852    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:39:41.416860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416858    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:39:41.417050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416863    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:39:41.417050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416869    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:39:41.417050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416875    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:39:41.417050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416884    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:39:41.417050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416892    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:39:41.417050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416915    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:39:41.417050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.416920    4282 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:39:41.420454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.420441    4282 server.go:1263] "Started kubelet"
Oct 30 21:39:41.420553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.420527    4282 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:39:41.421069 ip-10-0-131-245 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:39:41.421324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.421238    4282 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:39:41.421408 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.421333    4282 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:39:41.422083 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.422065    4282 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:39:41.422211 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.422199    4282 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.424850    4282 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:39:41.425853    4282 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-245.us-east-2.compute.internal\" not found"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.425880    4282 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.425894    4282 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.425902    4282 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.426141    4282 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427378    4282 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:34:04 +0000 UTC" deadline="2028-06-21 17:58:34.222121072 +0000 UTC"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427489    4282 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14396h18m52.794660233s"
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427512    4282 factory.go:55] Registering systemd factory
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427505    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a11c4675-a114-40e1-a796-2443b92f1218" volumeName="kubernetes.io/projected/a11c4675-a114-40e1-a796-2443b92f1218-kube-api-access-2plh5" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427534    4282 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427556    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5" volumeName="kubernetes.io/projected/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kube-api-access-zswqn" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427577    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df49b636-0a33-4254-9692-3bc1870f181d" volumeName="kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427595    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12528140-5996-44d5-b1ea-a1eea5b4a65c" volumeName="kubernetes.io/configmap/12528140-5996-44d5-b1ea-a1eea5b4a65c-serviceca" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427607    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25874bd6-56b8-4935-80eb-c9fcb240ec56" volumeName="kubernetes.io/configmap/25874bd6-56b8-4935-80eb-c9fcb240ec56-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427619    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4989e2a1-ce7d-4da1-b626-946742bf4897" volumeName="kubernetes.io/empty-dir/4989e2a1-ce7d-4da1-b626-946742bf4897-tmp-dir" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427635    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858066ef-e2da-42a9-9044-239570efa65c" volumeName="kubernetes.io/projected/858066ef-e2da-42a9-9044-239570efa65c-kube-api-access-ttknd" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427649    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9" volumeName="kubernetes.io/secret/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-agent-certs" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427669    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12528140-5996-44d5-b1ea-a1eea5b4a65c" volumeName="kubernetes.io/projected/12528140-5996-44d5-b1ea-a1eea5b4a65c-kube-api-access-s7cpd" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427681    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="133bfc6c-cab2-4a9c-8710-e87f73f73804" volumeName="kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-tmp" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427698    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858066ef-e2da-42a9-9044-239570efa65c" volumeName="kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427710    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858066ef-e2da-42a9-9044-239570efa65c" volumeName="kubernetes.io/secret/858066ef-e2da-42a9-9044-239570efa65c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427723    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a11c4675-a114-40e1-a796-2443b92f1218" volumeName="kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427744    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df49b636-0a33-4254-9692-3bc1870f181d" volumeName="kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427766    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df49b636-0a33-4254-9692-3bc1870f181d" volumeName="kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427787    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df49b636-0a33-4254-9692-3bc1870f181d" volumeName="kubernetes.io/projected/df49b636-0a33-4254-9692-3bc1870f181d-kube-api-access-xwl5t" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427799    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="133bfc6c-cab2-4a9c-8710-e87f73f73804" volumeName="kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-tuned" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427815    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25874bd6-56b8-4935-80eb-c9fcb240ec56" volumeName="kubernetes.io/projected/25874bd6-56b8-4935-80eb-c9fcb240ec56-kube-api-access-p6qbm" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427827    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58d0b21d-3085-4cd7-9989-68c046410d00" volumeName="kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427839    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858066ef-e2da-42a9-9044-239570efa65c" volumeName="kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-config" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427871    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a11c4675-a114-40e1-a796-2443b92f1218" volumeName="kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:39:41.427856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427883    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa1dbadd-b4a7-44cb-b35a-768e9315acfc" volumeName="kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret" seLinuxMountContext=""
Oct 30 21:39:41.429228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427900    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9" volumeName="kubernetes.io/configmap/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:39:41.429228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427914    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="133bfc6c-cab2-4a9c-8710-e87f73f73804" volumeName="kubernetes.io/projected/133bfc6c-cab2-4a9c-8710-e87f73f73804-kube-api-access-r9vfg" seLinuxMountContext=""
Oct 30 21:39:41.429228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427932    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4989e2a1-ce7d-4da1-b626-946742bf4897" volumeName="kubernetes.io/projected/4989e2a1-ce7d-4da1-b626-946742bf4897-kube-api-access-bnrb9" seLinuxMountContext=""
Oct 30 21:39:41.429228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427947    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858066ef-e2da-42a9-9044-239570efa65c" volumeName="kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-env-overrides" seLinuxMountContext=""
Oct 30 21:39:41.429228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427960    4282 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98efb6cf-f6f5-4932-af9c-04640d6f338a" volumeName="kubernetes.io/projected/98efb6cf-f6f5-4932-af9c-04640d6f338a-kube-api-access-jcxcg" seLinuxMountContext=""
Oct 30 21:39:41.429228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427972    4282 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:39:41.429228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.427980    4282 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:39:41.429597 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.429582    4282 factory.go:153] Registering CRI-O factory
Oct 30 21:39:41.429597 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.429598    4282 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:39:41.429759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.429649    4282 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:39:41.429759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.429669    4282 factory.go:103] Registering Raw factory
Oct 30 21:39:41.429759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.429685    4282 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:39:41.429949 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.429926    4282 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:41.430889 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.430865    4282 manager.go:319] Starting recovery of all containers
Oct 30 21:39:41.431093 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.431075    4282 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:41.432275 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.432225    4282 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:41.434021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.434004    4282 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:39:41.454348 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.454319    4282 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:39:41.455781 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.455759    4282 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:39:41.455781 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.455780    4282 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:39:41.455940 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.455798    4282 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:39:41.455940 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:39:41.455838    4282 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:39:41.458598 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.458582    4282 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:39:41.480714 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.480687    4282 generic.go:356] "Generic (PLEG): container finished" podID="0788c9cb659ae56a4649dcd1f52d9f30" containerID="55850a9a26e900617508720e39c7f9d84fe54c6e77c008ee9767e4c5dff762b5" exitCode=0
Oct 30 21:39:41.489222 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.489195    4282 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="ea34b14d32b90cdf2bf19068b06555bc18625bbe38286e86a1de6a120bbd4c04" exitCode=0
Oct 30 21:39:41.489222 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.489221    4282 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="18e5213e260edd8a41174f596197ed81a2b85b00260407b91f78e059cb7debdd" exitCode=0
Oct 30 21:39:41.489339 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.489231    4282 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="6491f7201c17c705f13d03eadbcaec745b60d6c3756b28fdcfb2323b79b374eb" exitCode=0
Oct 30 21:39:41.489339 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.489240    4282 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="d7245da925453a54b2431c77a9f8cb2f7909bc883395b2c8a3bacdb296b9f390" exitCode=0
Oct 30 21:39:41.489339 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.489249    4282 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="8b86f7a6bc1bf317c90e0908b6da5fa8cba36f95458beac93aaa119a6e63f83d" exitCode=0
Oct 30 21:39:41.489339 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.489255    4282 generic.go:356] "Generic (PLEG): container finished" podID="df49b636-0a33-4254-9692-3bc1870f181d" containerID="89d767db795f540365767a345abb4e98d21f93a293a0d179501be1dc1f0da3e1" exitCode=0
Oct 30 21:39:41.489339 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.489263    4282 manager.go:324] Recovery completed
Oct 30 21:39:41.501800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.501780    4282 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:39:41.501800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.501797    4282 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:39:41.501919 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.501817    4282 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:39:41.501952 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.501923    4282 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:39:41.501952 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.501934    4282 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:39:41.502007 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.501995    4282 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:39:41.502007 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.502001    4282 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:39:41.502007 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.502006    4282 policy_none.go:50] "Start"
Oct 30 21:39:41.502090 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.502014    4282 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:39:41.502090 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.502023    4282 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:39:41.502177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.502166    4282 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:39:41.502206 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.502178    4282 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:39:41.502206 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.502183    4282 policy_none.go:44] "Start"
Oct 30 21:39:41.505988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.505963    4282 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:39:41.508547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.508535    4282 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:39:41.509048 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:39:41.508558    4282 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:39:41.509048 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.509049    4282 server.go:87] "Starting device plugin registration server"
Oct 30 21:39:41.509296 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.509282    4282 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:39:41.509384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.509294    4282 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:39:41.509440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.509423    4282 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:39:41.509589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.509533    4282 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:39:41.509589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.509576    4282 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:39:41.509589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.509584    4282 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:39:41.509820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.509634    4282 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:39:41.509564601Z","UUID":"e48e15e8-b4ec-460c-bd7c-350c6c9fd294","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:39:41.510719 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:39:41.510701    4282 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:39:41.511060 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.511042    4282 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:39:41.511138 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.511079    4282 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:39:41.556948 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.556851    4282 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"]
Oct 30 21:39:41.557320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.557282    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"a1e9e5008f7a1caba4d99ed76ccdf2ff","Type":"ContainerStarted","Data":"2d5f4395e9e17c892f016d88891fea055b732d8ae7b1d2e9e8a29569be316ad2"}
Oct 30 21:39:41.557429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.557390    4282 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.557497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.557459    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"a1e9e5008f7a1caba4d99ed76ccdf2ff","Type":"ContainerStarted","Data":"5ee4e37dee4e258080d4f0b20df83689a870fde8f925bfc4b036e1b70b6d678b"}
Oct 30 21:39:41.557529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.557500    4282 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.557589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.557505    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"0788c9cb659ae56a4649dcd1f52d9f30","Type":"ContainerStarted","Data":"dcc7ef95a12557c54140644df06a870b1aafec0dab0fcce6619644993a9e8d6b"}
Oct 30 21:39:41.557632 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.557605    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"0788c9cb659ae56a4649dcd1f52d9f30","Type":"ContainerDied","Data":"55850a9a26e900617508720e39c7f9d84fe54c6e77c008ee9767e4c5dff762b5"}
Oct 30 21:39:41.557632 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.557620    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal" event={"ID":"0788c9cb659ae56a4649dcd1f52d9f30","Type":"ContainerStarted","Data":"3f8a4fb8e510a96424c84147f74ea9866cc531496de47bfe3dbcaf8d09abfb87"}
Oct 30 21:39:41.567702 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.567685    4282 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:39:41.567749 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:39:41.567728    4282 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.567808 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.567797    4282 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:39:41.567842 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:39:41.567834    4282 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.609771 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.609748    4282 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:39:41.610960 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.610945    4282 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:39:41.611024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.610976    4282 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:39:41.611024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.610991    4282 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-245.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:39:41.611024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.611018    4282 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.625597 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.625578    4282 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.625644 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.625625    4282 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.635274 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.635254    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1e9e5008f7a1caba4d99ed76ccdf2ff-config\") pod \"kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"a1e9e5008f7a1caba4d99ed76ccdf2ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.635340 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.635283    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.635340 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.635306    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.735672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.735651    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1e9e5008f7a1caba4d99ed76ccdf2ff-config\") pod \"kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"a1e9e5008f7a1caba4d99ed76ccdf2ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.735751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.735614    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1e9e5008f7a1caba4d99ed76ccdf2ff-config\") pod \"kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"a1e9e5008f7a1caba4d99ed76ccdf2ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.735751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.735726    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.735823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.735742    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.735823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.735787    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:41.735883 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:41.735825    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0788c9cb659ae56a4649dcd1f52d9f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" (UID: \"0788c9cb659ae56a4649dcd1f52d9f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:42.414638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.414607    4282 apiserver.go:52] "Watching apiserver"
Oct 30 21:39:42.423253 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.423226    4282 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:39:42.423613 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.423595    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s","openshift-network-operator/iptables-alerter-2qlwc","openshift-ovn-kubernetes/ovnkube-node-kxgqs","kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-kqkkd","openshift-image-registry/node-ca-kbfjc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-zn8lv","openshift-multus/network-metrics-daemon-tgzlk","openshift-network-diagnostics/network-check-target-8k7kg","kube-system/konnectivity-agent-sqkkh","openshift-dns/node-resolver-82lq5","openshift-multus/multus-wwgj9","kube-system/global-pull-secret-syncer-r5xwr"]
Oct 30 21:39:42.427511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.427485    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:39:42.427624 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.427547    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:39:42.427682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.427634    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pnbm4\""
Oct 30 21:39:42.427751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.427717    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:39:42.428068 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.427993    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:39:42.428068 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428027    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:39:42.428316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428084    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:39:42.428316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428275    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:39:42.428735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428535    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2pxp\""
Oct 30 21:39:42.428735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428606    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:39:42.428735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428615    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:39:42.428735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428706    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:39:42.428735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428724    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:39:42.429020 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428727    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:39:42.429020 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428798    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9c426\""
Oct 30 21:39:42.429020 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.428841    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:39:42.429138 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.429020    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6cdkd\""
Oct 30 21:39:42.429138 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.429037    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:39:42.429519 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.429455    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:39:42.429617 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.429598    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:39:42.429750 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.429735    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:42.430278 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.430261    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wpx2j\""
Oct 30 21:39:42.430426 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.430409    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:39:42.430532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.430460    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:39:42.431706 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.431689    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-scjcc\""
Oct 30 21:39:42.431787 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.431776    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pp8j7\""
Oct 30 21:39:42.432076 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.432065    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:39:42.432154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.432144    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mh2p\""
Oct 30 21:39:42.432559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.432530    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:39:42.432679 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.432608    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:39:42.432679 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.432646    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:39:42.432679 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.432668    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:39:42.432894 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.432874    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:39:42.433029 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433019    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n2fsk\""
Oct 30 21:39:42.433577 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433563    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:39:42.433647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433589    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:39:42.433719 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433662    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jbnx4\""
Oct 30 21:39:42.433719 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433710    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzrj6\""
Oct 30 21:39:42.433855 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433735    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggbv4\""
Oct 30 21:39:42.433855 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433739    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:39:42.433855 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433785    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:39:42.433855 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433833    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:39:42.433983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.433865    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:39:42.434341 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.434327    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:39:42.438501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438486    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-cni-binary-copy\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.438553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438513    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-kubelet\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.438553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438532    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.438608 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438558    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.438636 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438615    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/858066ef-e2da-42a9-9044-239570efa65c-ovn-node-metrics-cert\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.438667 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438638    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-socket-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.438667 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438656    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-run\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.438734 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438671    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-ovn\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.438734 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438700    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-netd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.438813 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438735    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-script-lib\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.438813 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438783    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.438892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438834    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-bin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.438892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438860    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-cni-binary-copy\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.438892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438865    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-var-lib-kubelet\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.438892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438887    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12528140-5996-44d5-b1ea-a1eea5b4a65c-serviceca\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:42.439037 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438902    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cpd\" (UniqueName: \"kubernetes.io/projected/12528140-5996-44d5-b1ea-a1eea5b4a65c-kube-api-access-s7cpd\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:42.439037 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438936    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-script-lib\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439037 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.438963    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-socket-dir-parent\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.439037 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439025    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25874bd6-56b8-4935-80eb-c9fcb240ec56-iptables-alerter-script\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:42.439037 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439032    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/858066ef-e2da-42a9-9044-239570efa65c-ovn-node-metrics-cert\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439067    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-etc-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439094    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-sys-fs\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.439268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439120    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysconfig\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.439268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439149    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-hostroot\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.439268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439193    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-var-lib-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439244    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439457 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439275    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/25874bd6-56b8-4935-80eb-c9fcb240ec56-iptables-alerter-script\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:42.439457 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439294    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttknd\" (UniqueName: \"kubernetes.io/projected/858066ef-e2da-42a9-9044-239570efa65c-kube-api-access-ttknd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439457 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439319    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-sys\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.439457 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439402    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vfg\" (UniqueName: \"kubernetes.io/projected/133bfc6c-cab2-4a9c-8710-e87f73f73804-kube-api-access-r9vfg\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.439457 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439404    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12528140-5996-44d5-b1ea-a1eea5b4a65c-serviceca\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:42.439457 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439436    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qbm\" (UniqueName: \"kubernetes.io/projected/25874bd6-56b8-4935-80eb-c9fcb240ec56-kube-api-access-p6qbm\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:42.439745 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439573    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-lib-modules\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.439745 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439609    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-multus\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.439745 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439650    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-etc-kubernetes\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.439745 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439696    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-kubelet\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439745 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439722    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-systemd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439753    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-log-socket\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439775    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-bin\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439797    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-conf\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.439980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439819    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.439980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439859    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-config\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439896    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-env-overrides\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.439980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439923    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-k8s-cni-cncf-io\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.439980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439965    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-conf-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.439991    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-registration-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440023    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-device-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440047    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-env-overrides\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440045    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/858066ef-e2da-42a9-9044-239570efa65c-ovnkube-config\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440071    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-host\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440087    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-netns\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440109    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-multus-certs\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440146    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-modprobe-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440171    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-tuned\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.440239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440208    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12528140-5996-44d5-b1ea-a1eea5b4a65c-host\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440253    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-tuned\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440273    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-system-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440298    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2plh5\" (UniqueName: \"kubernetes.io/projected/a11c4675-a114-40e1-a796-2443b92f1218-kube-api-access-2plh5\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440322    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-systemd-units\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440360    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440389    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-systemd\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440404    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440421    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-tmp\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440444    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-multus-daemon-config\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440504    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133bfc6c-cab2-4a9c-8710-e87f73f73804-tmp\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440516    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-netns\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.440558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440547    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswqn\" (UniqueName: \"kubernetes.io/projected/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kube-api-access-zswqn\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.440872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440599    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-node-log\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.440872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440603    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a11c4675-a114-40e1-a796-2443b92f1218-multus-daemon-config\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.440872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440647    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-cnibin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.440872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440663    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25874bd6-56b8-4935-80eb-c9fcb240ec56-host-slash\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:42.440872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440697    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-slash\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.440872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.440740    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-kubernetes\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.450928 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.450900    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vfg\" (UniqueName: \"kubernetes.io/projected/133bfc6c-cab2-4a9c-8710-e87f73f73804-kube-api-access-r9vfg\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.451733 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.451714    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttknd\" (UniqueName: \"kubernetes.io/projected/858066ef-e2da-42a9-9044-239570efa65c-kube-api-access-ttknd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.451815 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.451750    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2plh5\" (UniqueName: \"kubernetes.io/projected/a11c4675-a114-40e1-a796-2443b92f1218-kube-api-access-2plh5\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.451935 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.451911    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qbm\" (UniqueName: \"kubernetes.io/projected/25874bd6-56b8-4935-80eb-c9fcb240ec56-kube-api-access-p6qbm\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:42.451995 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.451917    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswqn\" (UniqueName: \"kubernetes.io/projected/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kube-api-access-zswqn\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.457685 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.457669    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cpd\" (UniqueName: \"kubernetes.io/projected/12528140-5996-44d5-b1ea-a1eea5b4a65c-kube-api-access-s7cpd\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:42.491452 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.491423    4282 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:42.491563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.491548    4282 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:42.502417 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.502401    4282 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:39:42.502532 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:39:42.502440    4282 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:42.503310 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.503293    4282 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:39:42.503376 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:39:42.503329    4282 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-245.us-east-2.compute.internal"
Oct 30 21:39:42.526899 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.526880    4282 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:39:42.541220 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541198    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-device-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.541302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541227    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-cnibin\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.541302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541244    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.541302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541258    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-host\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.541302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541273    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-netns\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541346    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-host\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541380    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-multus-certs\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541397    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4989e2a1-ce7d-4da1-b626-946742bf4897-tmp-dir\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541413    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-modprobe-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541428    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12528140-5996-44d5-b1ea-a1eea5b4a65c-host\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541455    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12528140-5996-44d5-b1ea-a1eea5b4a65c-host\") pod \"node-ca-kbfjc\" (UID: \"12528140-5996-44d5-b1ea-a1eea5b4a65c\") " pod="openshift-image-registry/node-ca-kbfjc"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541491    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-netns\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541497    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-multus-certs\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541516    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-device-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541530    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4989e2a1-ce7d-4da1-b626-946742bf4897-tmp-dir\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:42.541551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541535    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-system-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541570    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-systemd-units\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541587    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541589    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-system-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541592    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-systemd-units\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541609    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-systemd\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541636    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541653    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxcg\" (UniqueName: \"kubernetes.io/projected/98efb6cf-f6f5-4932-af9c-04640d6f338a-kube-api-access-jcxcg\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541570    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-modprobe-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541667    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541673    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-systemd\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541688    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-netns\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541704    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-etc-selinux\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541724    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-netns\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541723    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-node-log\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541740    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-node-log\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541748    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-binary-copy\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541775    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541791    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-kubelet-config\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541793    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-d\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541806    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-cnibin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541844    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-cnibin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541846    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.541978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.541907    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-binary-copy\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542233    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25874bd6-56b8-4935-80eb-c9fcb240ec56-host-slash\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542251    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-slash\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542268    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-system-cni-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542286    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-konnectivity-ca\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542312    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-kubernetes\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542334    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-slash\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542335    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-kubelet\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542367    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-kubernetes\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542365    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-kubelet\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542338    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/25874bd6-56b8-4935-80eb-c9fcb240ec56-host-slash\") pod \"iptables-alerter-2qlwc\" (UID: \"25874bd6-56b8-4935-80eb-c9fcb240ec56\") " pod="openshift-network-operator/iptables-alerter-2qlwc"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542374    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542397    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542413    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542432    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4989e2a1-ce7d-4da1-b626-946742bf4897-hosts-file\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542451    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-socket-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542456    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542492    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-run\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542516    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-ovn\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542538    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-netd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542536    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-socket-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542562    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542572    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-ovn\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542574    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-run\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542585    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542589    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-netd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542610    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542641    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-konnectivity-ca\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542653    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrb9\" (UniqueName: \"kubernetes.io/projected/4989e2a1-ce7d-4da1-b626-946742bf4897-kube-api-access-bnrb9\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542676    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-bin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.542717 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542690    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-var-lib-kubelet\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.542750    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-bin\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543012    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-socket-dir-parent\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543031    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/df49b636-0a33-4254-9692-3bc1870f181d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543088    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-socket-dir-parent\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543113    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-var-lib-kubelet\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543151    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-etc-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543185    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-sys-fs\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543214    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysconfig\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543231    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-etc-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543240    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-hostroot\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543300    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-sys-fs\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543364    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysconfig\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543371    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-hostroot\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543301    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-var-lib-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543407    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-var-lib-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543408    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543448    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-openvswitch\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543450    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543523    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-sys\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543589    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-dbus\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543627    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-lib-modules\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543659    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-multus\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543675    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-original-pull-secret\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543692    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwl5t\" (UniqueName: \"kubernetes.io/projected/df49b636-0a33-4254-9692-3bc1870f181d-kube-api-access-xwl5t\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543726    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-etc-kubernetes\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543760    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-kubelet\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543788    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-systemd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543826    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-log-socket\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543857    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-bin\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543881    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543909    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-agent-certs\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543987    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-log-socket\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543994    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-run-systemd\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.543573    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-sys\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544041    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-kubelet\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544070    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-etc-kubernetes\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.544084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544085    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-conf\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544147    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544210    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/858066ef-e2da-42a9-9044-239570efa65c-host-cni-bin\") pod \"ovnkube-node-kxgqs\" (UID: \"858066ef-e2da-42a9-9044-239570efa65c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544224    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-k8s-cni-cncf-io\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544275    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-cni-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544290    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-conf-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544321    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-var-lib-cni-multus\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544328    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-registration-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544295    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-host-run-k8s-cni-cncf-io\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544391    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-lib-modules\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544403    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5-registration-dir\") pod \"aws-ebs-csi-driver-node-9hm4s\" (UID: \"b0b646a1-f444-48b7-b8ee-5cfd2b87c7a5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9hm4s"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544450    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a11c4675-a114-40e1-a796-2443b92f1218-multus-conf-dir\") pod \"multus-wwgj9\" (UID: \"a11c4675-a114-40e1-a796-2443b92f1218\") " pod="openshift-multus/multus-wwgj9"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544499    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9-agent-certs\") pod \"konnectivity-agent-sqkkh\" (UID: \"cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9\") " pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:42.545655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.544523    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/133bfc6c-cab2-4a9c-8710-e87f73f73804-etc-sysctl-conf\") pod \"tuned-kqkkd\" (UID: \"133bfc6c-cab2-4a9c-8710-e87f73f73804\") " pod="openshift-cluster-node-tuning-operator/tuned-kqkkd"
Oct 30 21:39:42.550762 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.550739    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxcg\" (UniqueName: \"kubernetes.io/projected/98efb6cf-f6f5-4932-af9c-04640d6f338a-kube-api-access-jcxcg\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:42.551677 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.551662    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrb9\" (UniqueName: \"kubernetes.io/projected/4989e2a1-ce7d-4da1-b626-946742bf4897-kube-api-access-bnrb9\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:42.552240 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.552224    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwl5t\" (UniqueName: \"kubernetes.io/projected/df49b636-0a33-4254-9692-3bc1870f181d-kube-api-access-xwl5t\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.552830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.552811    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nf9\" (UniqueName: \"kubernetes.io/projected/58d0b21d-3085-4cd7-9989-68c046410d00-kube-api-access-k4nf9\") pod \"network-check-target-8k7kg\" (UID: \"58d0b21d-3085-4cd7-9989-68c046410d00\") " pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:42.644801 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644771    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-cnibin\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.644908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644806    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.644908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644836    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:42.644908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644856    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-kubelet-config\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:42.644908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644890    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-cnibin\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.645125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644906    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.645125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644914    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-kubelet-config\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:42.645125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644962    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-system-cni-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.645125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.644996    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4989e2a1-ce7d-4da1-b626-946742bf4897-hosts-file\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:42.645125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.645022    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/df49b636-0a33-4254-9692-3bc1870f181d-system-cni-dir\") pod \"multus-additional-cni-plugins-zn8lv\" (UID: \"df49b636-0a33-4254-9692-3bc1870f181d\") " pod="openshift-multus/multus-additional-cni-plugins-zn8lv"
Oct 30 21:39:42.645125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.645039    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-dbus\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:42.645125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.645071    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4989e2a1-ce7d-4da1-b626-946742bf4897-hosts-file\") pod \"node-resolver-82lq5\" (UID: \"4989e2a1-ce7d-4da1-b626-946742bf4897\") " pod="openshift-dns/node-resolver-82lq5"
Oct 30 21:39:42.645125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.645086    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aa1dbadd-b4a7-44cb-b35a-768e9315acfc-dbus\") pod \"global-pull-secret-syncer-r5xwr\" (UID: \"aa1dbadd-b4a7-44cb-b35a-768e9315acfc\") " pod="kube-system/global-pull-secret-syncer-r5xwr"
Oct 30 21:39:42.645376 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.645186    4282 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:39:42.647355 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.647340    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98efb6cf-f6f5-4932-af9c-04640d6f338a-metrics-certs\") pod \"network-metrics-daemon-tgzlk\" (UID: \"98efb6cf-f6f5-4932-af9c-04640d6f338a\") " pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:42.730693 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:42.730671    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgzlk"
Oct 30 21:39:43.045534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:43.045442    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tgzlk"]
Oct 30 21:39:43.052590 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:43.050360    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98efb6cf_f6f5_4932_af9c_04640d6f338a.slice/crio-e8955a1a251153723ab9becbdccfcf683eafe7d70096394657a051bf125bddab WatchSource:0}: Error finding container e8955a1a251153723ab9becbdccfcf683eafe7d70096394657a051bf125bddab: Status 404 returned error can't find the container with id e8955a1a251153723ab9becbdccfcf683eafe7d70096394657a051bf125bddab
Oct 30 21:39:43.054005 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:43.053975    4282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:39:43.494424 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:43.494392    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgzlk" event={"ID":"98efb6cf-f6f5-4932-af9c-04640d6f338a","Type":"ContainerStarted","Data":"e8955a1a251153723ab9becbdccfcf683eafe7d70096394657a051bf125bddab"}
Oct 30 21:39:44.687270 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:44.687246    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:45.499949 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:45.499913    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgzlk" event={"ID":"98efb6cf-f6f5-4932-af9c-04640d6f338a","Type":"ContainerStarted","Data":"6868a0fd6d14b72630d99a40d94aebf53ddd42387c56cac74768921269a53b71"}
Oct 30 21:39:45.500158 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:45.500000    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgzlk" event={"ID":"98efb6cf-f6f5-4932-af9c-04640d6f338a","Type":"ContainerStarted","Data":"70bc2525c003216f244428bbc4e53c0a9b2aa5a59e255d920cfc01ec046bbcc2"}
Oct 30 21:39:48.598198 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:48.598159    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:39:49.825147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:49.825118    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sqkkh"
Oct 30 21:39:50.394207 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.394176    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:39:50.394450 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.394433    4282 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:39:50.518329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.518297    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-5m9gv-6wpk2"]
Oct 30 21:39:50.522927 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.522901    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.533417 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.533396    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8td58\""
Oct 30 21:39:50.534294 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.534266    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5m9gv-6wpk2"]
Oct 30 21:39:50.534840 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.534821    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"]
Oct 30 21:39:50.536735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.536715    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:39:50.537005 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.536986    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:39:50.537356 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.537336    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:39:50.537496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.537456    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"
Oct 30 21:39:50.538028 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.538011    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:39:50.560244 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.560221    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"]
Oct 30 21:39:50.577617 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.577596    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-84r6w"]
Oct 30 21:39:50.580104 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.580088    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-84r6w"
Oct 30 21:39:50.587419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.587394    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szr7\" (UniqueName: \"kubernetes.io/projected/2f74abea-1ffe-482b-bc0f-31d297aca1f8-kube-api-access-7szr7\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.587516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.587433    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f74abea-1ffe-482b-bc0f-31d297aca1f8-archives-path\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.587516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.587456    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-service-ca-bundle\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.587612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.587526    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-trusted-ca-bundle\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.587612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.587607    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/385e7e99-d468-4345-b432-dd1565ef5c5c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t4fkx\" (UID: \"385e7e99-d468-4345-b432-dd1565ef5c5c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"
Oct 30 21:39:50.596284 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.596253    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-84r6w"]
Oct 30 21:39:50.596738 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.596717    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9l44m"]
Oct 30 21:39:50.599792 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.599771    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.608596 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.608580    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:39:50.608765 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.608751    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:39:50.618937 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.618923    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:39:50.619342 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.619328    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9l44m"]
Oct 30 21:39:50.645878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.645828    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 21:39:50.655308 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.655291    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2ktl4\""
Oct 30 21:39:50.655360 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.655326    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-csxns\""
Oct 30 21:39:50.655391 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.655365    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45g4h\""
Oct 30 21:39:50.655391 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.655375    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:39:50.655448 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.655404    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:39:50.687875 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.687853    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f74abea-1ffe-482b-bc0f-31d297aca1f8-archives-path\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.687982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.687884    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.687982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.687903    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w9md\" (UniqueName: \"kubernetes.io/projected/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-kube-api-access-8w9md\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.687982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.687921    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-service-ca-bundle\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.687982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.687937    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-trusted-ca-bundle\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.688133 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.688003    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/385e7e99-d468-4345-b432-dd1565ef5c5c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t4fkx\" (UID: \"385e7e99-d468-4345-b432-dd1565ef5c5c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"
Oct 30 21:39:50.688133 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.688038    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.688133 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.688090    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1010c6-da78-4b7f-ac5c-9c86e570750d-cert\") pod \"ingress-canary-84r6w\" (UID: \"cd1010c6-da78-4b7f-ac5c-9c86e570750d\") " pod="openshift-ingress-canary/ingress-canary-84r6w"
Oct 30 21:39:50.688280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.688200    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szr7\" (UniqueName: \"kubernetes.io/projected/2f74abea-1ffe-482b-bc0f-31d297aca1f8-kube-api-access-7szr7\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.688280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.688239    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-crio-socket\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.688391 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.688313    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-data-volume\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.688443 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.688391    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f889h\" (UniqueName: \"kubernetes.io/projected/cd1010c6-da78-4b7f-ac5c-9c86e570750d-kube-api-access-f889h\") pod \"ingress-canary-84r6w\" (UID: \"cd1010c6-da78-4b7f-ac5c-9c86e570750d\") " pod="openshift-ingress-canary/ingress-canary-84r6w"
Oct 30 21:39:50.688585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.688558    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f74abea-1ffe-482b-bc0f-31d297aca1f8-archives-path\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.689026 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.689002    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-service-ca-bundle\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.689172 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.689153    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-trusted-ca-bundle\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.689252 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.689182    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-vp2lj"]
Oct 30 21:39:50.693205 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.693121    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:39:50.694045 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.694025    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/385e7e99-d468-4345-b432-dd1565ef5c5c-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-t4fkx\" (UID: \"385e7e99-d468-4345-b432-dd1565ef5c5c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"
Oct 30 21:39:50.706204 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.706183    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6886d97769-h7tkk"]
Oct 30 21:39:50.708801 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.708785    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.725892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.725866    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:39:50.725988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.725973    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:39:50.730520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.730494    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:39:50.730991 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.730973    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vp2lj"]
Oct 30 21:39:50.737363 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.737323    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:39:50.737363 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.737325    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bk7d4\""
Oct 30 21:39:50.737536 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.737365    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:39:50.737536 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.737393    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:39:50.737536 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.737417    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l6nmx\""
Oct 30 21:39:50.740195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.740180    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szr7\" (UniqueName: \"kubernetes.io/projected/2f74abea-1ffe-482b-bc0f-31d297aca1f8-kube-api-access-7szr7\") pod \"periodic-gathering-5m9gv-6wpk2\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") " pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.760028 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.760005    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-764zc"]
Oct 30 21:39:50.763170 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.763151    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6886d97769-h7tkk"]
Oct 30 21:39:50.763273 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.763251    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.778195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.778177    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:39:50.778918 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.778892    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-764zc"]
Oct 30 21:39:50.788838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.788817    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed9aa114-485b-4f90-b18e-f3bfb0493287-image-registry-private-configuration\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.788925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.788845    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-registry-tls\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.788925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.788867    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.788925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.788882    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02da34f8-540c-4b70-a9d3-6cf004c05a10-config-volume\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.789050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789038    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed9aa114-485b-4f90-b18e-f3bfb0493287-ca-trust-extracted\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.789082 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789060    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w9md\" (UniqueName: \"kubernetes.io/projected/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-kube-api-access-8w9md\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.789082 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789074    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51815808-5f76-4970-a434-88c3e11b2348-tmp\") pod \"downloads-6dd789c8b7-vp2lj\" (UID: \"51815808-5f76-4970-a434-88c3e11b2348\") " pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:39:50.789148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789095    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmsg\" (UniqueName: \"kubernetes.io/projected/02da34f8-540c-4b70-a9d3-6cf004c05a10-kube-api-access-sgmsg\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.789148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789119    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.789248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789206    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed9aa114-485b-4f90-b18e-f3bfb0493287-registry-certificates\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.789248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789227    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed9aa114-485b-4f90-b18e-f3bfb0493287-trusted-ca\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.789248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789242    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02da34f8-540c-4b70-a9d3-6cf004c05a10-metrics-tls\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.789395 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789258    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbqt\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-kube-api-access-vhbqt\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.789395 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789371    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1010c6-da78-4b7f-ac5c-9c86e570750d-cert\") pod \"ingress-canary-84r6w\" (UID: \"cd1010c6-da78-4b7f-ac5c-9c86e570750d\") " pod="openshift-ingress-canary/ingress-canary-84r6w"
Oct 30 21:39:50.789529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789512    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-crio-socket\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.789589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789537    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-data-volume\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.789589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789562    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed9aa114-485b-4f90-b18e-f3bfb0493287-installation-pull-secrets\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.789694 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789596    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-crio-socket\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.789694 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789598    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gfpk\" (UniqueName: \"kubernetes.io/projected/51815808-5f76-4970-a434-88c3e11b2348-kube-api-access-7gfpk\") pod \"downloads-6dd789c8b7-vp2lj\" (UID: \"51815808-5f76-4970-a434-88c3e11b2348\") " pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:39:50.789694 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789674    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f889h\" (UniqueName: \"kubernetes.io/projected/cd1010c6-da78-4b7f-ac5c-9c86e570750d-kube-api-access-f889h\") pod \"ingress-canary-84r6w\" (UID: \"cd1010c6-da78-4b7f-ac5c-9c86e570750d\") " pod="openshift-ingress-canary/ingress-canary-84r6w"
Oct 30 21:39:50.789837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789726    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02da34f8-540c-4b70-a9d3-6cf004c05a10-tmp-dir\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.789837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789736    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.789837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.789752    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-bound-sa-token\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.790043 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.790017    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-data-volume\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.791505 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.791486    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.791974 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.791952    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd1010c6-da78-4b7f-ac5c-9c86e570750d-cert\") pod \"ingress-canary-84r6w\" (UID: \"cd1010c6-da78-4b7f-ac5c-9c86e570750d\") " pod="openshift-ingress-canary/ingress-canary-84r6w"
Oct 30 21:39:50.805124 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.805093    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f889h\" (UniqueName: \"kubernetes.io/projected/cd1010c6-da78-4b7f-ac5c-9c86e570750d-kube-api-access-f889h\") pod \"ingress-canary-84r6w\" (UID: \"cd1010c6-da78-4b7f-ac5c-9c86e570750d\") " pod="openshift-ingress-canary/ingress-canary-84r6w"
Oct 30 21:39:50.805520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.805504    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w9md\" (UniqueName: \"kubernetes.io/projected/fb3f54d0-c509-4b9f-85f6-734d6756b9fc-kube-api-access-8w9md\") pod \"insights-runtime-extractor-9l44m\" (UID: \"fb3f54d0-c509-4b9f-85f6-734d6756b9fc\") " pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.833791 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.833767    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:39:50.836146 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.836128    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-glzqf\""
Oct 30 21:39:50.836210 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.836128    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:39:50.847722 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.847704    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"
Oct 30 21:39:50.890003 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.889973    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-84r6w"
Oct 30 21:39:50.891753 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891728    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02da34f8-540c-4b70-a9d3-6cf004c05a10-tmp-dir\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.891882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891767    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-bound-sa-token\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.891882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891801    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed9aa114-485b-4f90-b18e-f3bfb0493287-image-registry-private-configuration\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.891882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891829    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-registry-tls\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.891882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891861    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02da34f8-540c-4b70-a9d3-6cf004c05a10-config-volume\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.892073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891886    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed9aa114-485b-4f90-b18e-f3bfb0493287-ca-trust-extracted\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.892073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891915    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51815808-5f76-4970-a434-88c3e11b2348-tmp\") pod \"downloads-6dd789c8b7-vp2lj\" (UID: \"51815808-5f76-4970-a434-88c3e11b2348\") " pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:39:50.892073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891947    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmsg\" (UniqueName: \"kubernetes.io/projected/02da34f8-540c-4b70-a9d3-6cf004c05a10-kube-api-access-sgmsg\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.892073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.891990    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed9aa114-485b-4f90-b18e-f3bfb0493287-registry-certificates\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.892073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.892013    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed9aa114-485b-4f90-b18e-f3bfb0493287-trusted-ca\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.892073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.892039    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02da34f8-540c-4b70-a9d3-6cf004c05a10-metrics-tls\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.892073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.892064    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbqt\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-kube-api-access-vhbqt\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.892399 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.892102    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed9aa114-485b-4f90-b18e-f3bfb0493287-installation-pull-secrets\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.892399 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.892130    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gfpk\" (UniqueName: \"kubernetes.io/projected/51815808-5f76-4970-a434-88c3e11b2348-kube-api-access-7gfpk\") pod \"downloads-6dd789c8b7-vp2lj\" (UID: \"51815808-5f76-4970-a434-88c3e11b2348\") " pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:39:50.892399 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.892142    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02da34f8-540c-4b70-a9d3-6cf004c05a10-tmp-dir\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.892809 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.892789    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ed9aa114-485b-4f90-b18e-f3bfb0493287-ca-trust-extracted\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.892911 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.892889    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02da34f8-540c-4b70-a9d3-6cf004c05a10-config-volume\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.894618 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.893145    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51815808-5f76-4970-a434-88c3e11b2348-tmp\") pod \"downloads-6dd789c8b7-vp2lj\" (UID: \"51815808-5f76-4970-a434-88c3e11b2348\") " pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:39:50.894618 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.894429    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ed9aa114-485b-4f90-b18e-f3bfb0493287-registry-certificates\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.894618 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.894543    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ed9aa114-485b-4f90-b18e-f3bfb0493287-trusted-ca\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.898523 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.897950    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02da34f8-540c-4b70-a9d3-6cf004c05a10-metrics-tls\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:50.898523 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.897979    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ed9aa114-485b-4f90-b18e-f3bfb0493287-image-registry-private-configuration\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.898523 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.898187    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ed9aa114-485b-4f90-b18e-f3bfb0493287-installation-pull-secrets\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.898728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.898641    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-registry-tls\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.906844 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.906483    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-bound-sa-token\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.930148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.909913    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbqt\" (UniqueName: \"kubernetes.io/projected/ed9aa114-485b-4f90-b18e-f3bfb0493287-kube-api-access-vhbqt\") pod \"image-registry-6886d97769-h7tkk\" (UID: \"ed9aa114-485b-4f90-b18e-f3bfb0493287\") " pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:50.930148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.914976    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gfpk\" (UniqueName: \"kubernetes.io/projected/51815808-5f76-4970-a434-88c3e11b2348-kube-api-access-7gfpk\") pod \"downloads-6dd789c8b7-vp2lj\" (UID: \"51815808-5f76-4970-a434-88c3e11b2348\") " pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:39:50.930148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.915220    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9l44m"
Oct 30 21:39:50.930148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:50.929558    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmsg\" (UniqueName: \"kubernetes.io/projected/02da34f8-540c-4b70-a9d3-6cf004c05a10-kube-api-access-sgmsg\") pod \"dns-default-764zc\" (UID: \"02da34f8-540c-4b70-a9d3-6cf004c05a10\") " pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:51.006220 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.005860    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:39:51.019042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.019014    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:51.076549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.076433    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:51.112887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.112812    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-5m9gv-6wpk2"]
Oct 30 21:39:51.117802 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:51.117774    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f74abea_1ffe_482b_bc0f_31d297aca1f8.slice/crio-33368d94faa2393d6fd9c9de872c50ecc92b2412e9e5d0e970e2a55e5b43e50c WatchSource:0}: Error finding container 33368d94faa2393d6fd9c9de872c50ecc92b2412e9e5d0e970e2a55e5b43e50c: Status 404 returned error can't find the container with id 33368d94faa2393d6fd9c9de872c50ecc92b2412e9e5d0e970e2a55e5b43e50c
Oct 30 21:39:51.134212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.133982    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"]
Oct 30 21:39:51.139847 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:51.139804    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385e7e99_d468_4345_b432_dd1565ef5c5c.slice/crio-ee174698284351487a8e2f224a15f5764a79b4c38adbf967de5bcaea7aa62561 WatchSource:0}: Error finding container ee174698284351487a8e2f224a15f5764a79b4c38adbf967de5bcaea7aa62561: Status 404 returned error can't find the container with id ee174698284351487a8e2f224a15f5764a79b4c38adbf967de5bcaea7aa62561
Oct 30 21:39:51.201124 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.201087    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-84r6w"]
Oct 30 21:39:51.205229 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:51.205205    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1010c6_da78_4b7f_ac5c_9c86e570750d.slice/crio-6b96e3a8e7a9b3e7fdc5db03dbd10cb2ecc66fed9cf0f21cf9c346124764b887 WatchSource:0}: Error finding container 6b96e3a8e7a9b3e7fdc5db03dbd10cb2ecc66fed9cf0f21cf9c346124764b887: Status 404 returned error can't find the container with id 6b96e3a8e7a9b3e7fdc5db03dbd10cb2ecc66fed9cf0f21cf9c346124764b887
Oct 30 21:39:51.225435 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.225410    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9l44m"]
Oct 30 21:39:51.229184 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:51.229160    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3f54d0_c509_4b9f_85f6_734d6756b9fc.slice/crio-65303200e3046cb141a0b4fe54b5d72a0328636b50d2a61a84196d512be8bf4f WatchSource:0}: Error finding container 65303200e3046cb141a0b4fe54b5d72a0328636b50d2a61a84196d512be8bf4f: Status 404 returned error can't find the container with id 65303200e3046cb141a0b4fe54b5d72a0328636b50d2a61a84196d512be8bf4f
Oct 30 21:39:51.283443 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.283237    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-vp2lj"]
Oct 30 21:39:51.288252 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:51.288228    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51815808_5f76_4970_a434_88c3e11b2348.slice/crio-2960271f154f635235f6368f7e9e6ba1de84b5f414ffec746a54fc604ba41675 WatchSource:0}: Error finding container 2960271f154f635235f6368f7e9e6ba1de84b5f414ffec746a54fc604ba41675: Status 404 returned error can't find the container with id 2960271f154f635235f6368f7e9e6ba1de84b5f414ffec746a54fc604ba41675
Oct 30 21:39:51.310706 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.310674    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6886d97769-h7tkk"]
Oct 30 21:39:51.314287 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:51.314261    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9aa114_485b_4f90_b18e_f3bfb0493287.slice/crio-012105e183d20406cfb865a298085a9ca6b050164b362f5121804a3fd6244b4d WatchSource:0}: Error finding container 012105e183d20406cfb865a298085a9ca6b050164b362f5121804a3fd6244b4d: Status 404 returned error can't find the container with id 012105e183d20406cfb865a298085a9ca6b050164b362f5121804a3fd6244b4d
Oct 30 21:39:51.354861 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.354837    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-764zc"]
Oct 30 21:39:51.358381 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:39:51.358356    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02da34f8_540c_4b70_a9d3_6cf004c05a10.slice/crio-55a22279cbafe92554bd0d33a83233a7a630456ad4d08a70b5b1056603ae53ed WatchSource:0}: Error finding container 55a22279cbafe92554bd0d33a83233a7a630456ad4d08a70b5b1056603ae53ed: Status 404 returned error can't find the container with id 55a22279cbafe92554bd0d33a83233a7a630456ad4d08a70b5b1056603ae53ed
Oct 30 21:39:51.514384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.514305    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-764zc" event={"ID":"02da34f8-540c-4b70-a9d3-6cf004c05a10","Type":"ContainerStarted","Data":"55a22279cbafe92554bd0d33a83233a7a630456ad4d08a70b5b1056603ae53ed"}
Oct 30 21:39:51.515827 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.515799    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6886d97769-h7tkk" event={"ID":"ed9aa114-485b-4f90-b18e-f3bfb0493287","Type":"ContainerStarted","Data":"e8d2eaf9e7a7bed312426c496b48a50efa17e1589da8fd5a60307948d1b2ea43"}
Oct 30 21:39:51.515943 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.515845    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6886d97769-h7tkk" event={"ID":"ed9aa114-485b-4f90-b18e-f3bfb0493287","Type":"ContainerStarted","Data":"012105e183d20406cfb865a298085a9ca6b050164b362f5121804a3fd6244b4d"}
Oct 30 21:39:51.516996 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.516972    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-84r6w" event={"ID":"cd1010c6-da78-4b7f-ac5c-9c86e570750d","Type":"ContainerStarted","Data":"6b96e3a8e7a9b3e7fdc5db03dbd10cb2ecc66fed9cf0f21cf9c346124764b887"}
Oct 30 21:39:51.518075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.518055    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vp2lj" event={"ID":"51815808-5f76-4970-a434-88c3e11b2348","Type":"ContainerStarted","Data":"2960271f154f635235f6368f7e9e6ba1de84b5f414ffec746a54fc604ba41675"}
Oct 30 21:39:51.519751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.519732    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9l44m" event={"ID":"fb3f54d0-c509-4b9f-85f6-734d6756b9fc","Type":"ContainerStarted","Data":"3b3e3422a39cada86d0b52b80da9357025d677fc11e655c21da3ec2e5f37c8a1"}
Oct 30 21:39:51.519847 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.519801    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9l44m" event={"ID":"fb3f54d0-c509-4b9f-85f6-734d6756b9fc","Type":"ContainerStarted","Data":"65303200e3046cb141a0b4fe54b5d72a0328636b50d2a61a84196d512be8bf4f"}
Oct 30 21:39:51.520980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.520963    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx" event={"ID":"385e7e99-d468-4345-b432-dd1565ef5c5c","Type":"ContainerStarted","Data":"ee174698284351487a8e2f224a15f5764a79b4c38adbf967de5bcaea7aa62561"}
Oct 30 21:39:51.522111 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.522087    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2" event={"ID":"2f74abea-1ffe-482b-bc0f-31d297aca1f8","Type":"ContainerStarted","Data":"33368d94faa2393d6fd9c9de872c50ecc92b2412e9e5d0e970e2a55e5b43e50c"}
Oct 30 21:39:51.537887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:51.537848    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6886d97769-h7tkk" podStartSLOduration=1.537838721 podStartE2EDuration="1.537838721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:51.537129146 +0000 UTC m=+10.195225380" watchObservedRunningTime="2026-10-30 21:39:51.537838721 +0000 UTC m=+10.195934940"
Oct 30 21:39:52.539398 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:52.539308    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9l44m" event={"ID":"fb3f54d0-c509-4b9f-85f6-734d6756b9fc","Type":"ContainerStarted","Data":"71906e991cbb3bc873adaaca33b5a73a3e8c640a0728e3ad25c39215848a26e9"}
Oct 30 21:39:52.539852 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:52.539494    4282 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:39:53.545831 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:53.545788    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx" event={"ID":"385e7e99-d468-4345-b432-dd1565ef5c5c","Type":"ContainerStarted","Data":"f71e4f52e130d88b67e4f74cb5957f169e9c37e3bf4912fa6921ae6b50767b7d"}
Oct 30 21:39:54.554163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:54.553324    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-764zc" event={"ID":"02da34f8-540c-4b70-a9d3-6cf004c05a10","Type":"ContainerStarted","Data":"5dad590c92f0c02ee039e770989265562aac4b5c5b6d4a4f6ad2ed8f71c83b8d"}
Oct 30 21:39:54.554163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:54.553391    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-764zc" event={"ID":"02da34f8-540c-4b70-a9d3-6cf004c05a10","Type":"ContainerStarted","Data":"6dcdd625b38d5bbeea73d565dd3f6b6f4dba670ff5518a449b2fb58d13f13dfb"}
Oct 30 21:39:54.554163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:54.554109    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-764zc"
Oct 30 21:39:54.557077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:54.556256    4282 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:39:54.557077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:54.557008    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2" event={"ID":"2f74abea-1ffe-482b-bc0f-31d297aca1f8","Type":"ContainerStarted","Data":"ccb31c31ffdcd4d11fb190d3eed7309c41d2d397f00acfb46b57f957c8647d82"}
Oct 30 21:39:54.581902 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:54.581643    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx" podStartSLOduration=3.083740983 podStartE2EDuration="4.581627446s" totalImagesPullingTime="1.497886463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:53.562954131 +0000 UTC m=+12.221050359" watchObservedRunningTime="2026-10-30 21:39:54.581627446 +0000 UTC m=+13.239723667"
Oct 30 21:39:54.599376 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:54.599085    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2" podStartSLOduration=1.472277435 podStartE2EDuration="4.599069322s" totalImagesPullingTime="3.126791887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:54.59877813 +0000 UTC m=+13.256874349" watchObservedRunningTime="2026-10-30 21:39:54.599069322 +0000 UTC m=+13.257165547"
Oct 30 21:39:54.599376 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:54.599359    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-764zc" podStartSLOduration=1.7180532880000001 podStartE2EDuration="4.599351209s" totalImagesPullingTime="2.881297921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:54.582135413 +0000 UTC m=+13.240231634" watchObservedRunningTime="2026-10-30 21:39:54.599351209 +0000 UTC m=+13.257447426"
Oct 30 21:39:55.007916 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:55.007874    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:39:55.561748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:55.561709    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-84r6w" event={"ID":"cd1010c6-da78-4b7f-ac5c-9c86e570750d","Type":"ContainerStarted","Data":"a04effcb31b56e1fcbb6915b95ea623ac211dd533eed67dc70f941213cab3832"}
Oct 30 21:39:55.568181 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:55.568146    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9l44m" event={"ID":"fb3f54d0-c509-4b9f-85f6-734d6756b9fc","Type":"ContainerStarted","Data":"dd000c3ea5b38313d5af536cfc95d9954a878f5e0c18673e83076c19a17fed0b"}
Oct 30 21:39:55.579918 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:55.579867    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-84r6w" podStartSLOduration=1.651023581 podStartE2EDuration="5.579853026s" totalImagesPullingTime="3.928829445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:55.579635278 +0000 UTC m=+14.237731502" watchObservedRunningTime="2026-10-30 21:39:55.579853026 +0000 UTC m=+14.237949251"
Oct 30 21:39:55.599128 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:55.599074    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9l44m" podStartSLOduration=2.494466723 podStartE2EDuration="5.599058971s" totalImagesPullingTime="3.104592248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:55.597624513 +0000 UTC m=+14.255720737" watchObservedRunningTime="2026-10-30 21:39:55.599058971 +0000 UTC m=+14.257155195"
Oct 30 21:39:56.312134 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:56.312092    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-764zc_02da34f8-540c-4b70-a9d3-6cf004c05a10/dns/0.log"
Oct 30 21:39:56.477897 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:56.477862    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-764zc_02da34f8-540c-4b70-a9d3-6cf004c05a10/kube-rbac-proxy/0.log"
Oct 30 21:39:57.076365 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:57.076338    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-82lq5_4989e2a1-ce7d-4da1-b626-946742bf4897/dns-node-resolver/0.log"
Oct 30 21:39:57.677135 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:57.677103    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6886d97769-h7tkk_ed9aa114-485b-4f90-b18e-f3bfb0493287/registry/0.log"
Oct 30 21:39:57.946592 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:57.946518    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"
Oct 30 21:39:57.946745 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:57.946631    4282 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:39:57.952858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:57.952831    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-t4fkx"
Oct 30 21:39:58.276116 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:58.276045    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kbfjc_12528140-5996-44d5-b1ea-a1eea5b4a65c/node-ca/0.log"
Oct 30 21:39:59.077562 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:39:59.077536    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-84r6w_cd1010c6-da78-4b7f-ac5c-9c86e570750d/serve-healthcheck-canary/0.log"
Oct 30 21:40:03.997786 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:03.997749    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jcp77"]
Oct 30 21:40:04.002233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.002208    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.010859 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.010835    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:40:04.011188 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.011160    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:40:04.011405 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.011381    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9f5cm\""
Oct 30 21:40:04.011405 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.011381    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:40:04.011579 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.011426    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:40:04.011579 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.011433    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:40:04.011579 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.011428    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:40:04.095389 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095353    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdd6\" (UniqueName: \"kubernetes.io/projected/10f05285-32c1-4416-82fd-dd2d81a3a8bb-kube-api-access-vfdd6\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.095578 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095404    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10f05285-32c1-4416-82fd-dd2d81a3a8bb-metrics-client-ca\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.095578 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095516    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-sys\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.095578 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095566    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.095715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095668    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-root\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.095715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095695    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-tls\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.095813 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095721    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-wtmp\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.095865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095842    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-textfile\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.095921 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.095880    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.196694 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196646    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-wtmp\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.196881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196727    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-textfile\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.196881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196755    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.196881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196785    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdd6\" (UniqueName: \"kubernetes.io/projected/10f05285-32c1-4416-82fd-dd2d81a3a8bb-kube-api-access-vfdd6\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.196881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196816    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10f05285-32c1-4416-82fd-dd2d81a3a8bb-metrics-client-ca\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.196881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196847    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-sys\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.196881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196859    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-wtmp\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.196881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196875    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.197237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196938    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-root\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.197237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.196962    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-tls\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.197237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.197145    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-textfile\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.197237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.197212    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-sys\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.197237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.197215    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/10f05285-32c1-4416-82fd-dd2d81a3a8bb-root\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.197576 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.197537    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.197576 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.197546    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/10f05285-32c1-4416-82fd-dd2d81a3a8bb-metrics-client-ca\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.199747 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.199727    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.199876 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.199849    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/10f05285-32c1-4416-82fd-dd2d81a3a8bb-node-exporter-tls\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.209456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.209433    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdd6\" (UniqueName: \"kubernetes.io/projected/10f05285-32c1-4416-82fd-dd2d81a3a8bb-kube-api-access-vfdd6\") pod \"node-exporter-jcp77\" (UID: \"10f05285-32c1-4416-82fd-dd2d81a3a8bb\") " pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.319735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.319657    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jcp77"
Oct 30 21:40:04.333034 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:40:04.332999    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f05285_32c1_4416_82fd_dd2d81a3a8bb.slice/crio-bb293b4ca2e7336b471e20f4d46f43f0114b57bb94b1bc634d061f48229be980 WatchSource:0}: Error finding container bb293b4ca2e7336b471e20f4d46f43f0114b57bb94b1bc634d061f48229be980: Status 404 returned error can't find the container with id bb293b4ca2e7336b471e20f4d46f43f0114b57bb94b1bc634d061f48229be980
Oct 30 21:40:04.600269 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:04.600185    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jcp77" event={"ID":"10f05285-32c1-4416-82fd-dd2d81a3a8bb","Type":"ContainerStarted","Data":"bb293b4ca2e7336b471e20f4d46f43f0114b57bb94b1bc634d061f48229be980"}
Oct 30 21:40:04.650334 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:04.650289    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:40:05.055401 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.055370    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:05.059925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.059894    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.068174 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068130    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:40:05.068315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068223    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:40:05.068315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068293    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:40:05.068424 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068390    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:40:05.068737 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068483    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:40:05.068737 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068547    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:40:05.068737 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068606    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:40:05.068737 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068666    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:40:05.068937 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068754    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:40:05.068937 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.068854    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g8tpb\""
Oct 30 21:40:05.083486 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.083399    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:05.104162 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104128    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104162 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104157    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4x5w\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-kube-api-access-t4x5w\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104353 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104177    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-config-out\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104353 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104201    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104353 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104302    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104353 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104334    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104364    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104380    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104462    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104528    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104547    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104563    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-web-config\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.104593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.104584    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205449 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205411    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205449 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205457    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205506    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205537    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205564    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205586    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-web-config\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205618    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205643    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205665    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4x5w\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-kube-api-access-t4x5w\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205691    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-config-out\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.205711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205707    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.206143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205736    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.206143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.205783    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.207231 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.207198    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.207817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.207796    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.208929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.208901    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.209029 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.208908    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.209424 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.209367    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.211586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.211559    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-config-out\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.211813 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.211774    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.211887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.211808    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.211926 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.211879    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.211964 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.211928    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.213205 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.213186    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.214095 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.214075    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-web-config\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.218554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.218519    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4x5w\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-kube-api-access-t4x5w\") pod \"alertmanager-main-0\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.388402 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.388380    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:05.574415 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.574380    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-764zc"
Oct 30 21:40:05.605807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.605741    4282 generic.go:356] "Generic (PLEG): container finished" podID="10f05285-32c1-4416-82fd-dd2d81a3a8bb" containerID="66d4234b01152c35022134108f6e7a510cb165a84d5cee6bee65f20b82337721" exitCode=0
Oct 30 21:40:05.605926 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.605826    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jcp77" event={"ID":"10f05285-32c1-4416-82fd-dd2d81a3a8bb","Type":"ContainerDied","Data":"66d4234b01152c35022134108f6e7a510cb165a84d5cee6bee65f20b82337721"}
Oct 30 21:40:05.690293 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:05.690260    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:05.697545 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:40:05.697518    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36542c81_5234_4762_805e_29d56166bd2b.slice/crio-96eb309f66bddfb6092e1fc26d14ac7c8569dd53b48e1ec4ed216df0dfeeea61 WatchSource:0}: Error finding container 96eb309f66bddfb6092e1fc26d14ac7c8569dd53b48e1ec4ed216df0dfeeea61: Status 404 returned error can't find the container with id 96eb309f66bddfb6092e1fc26d14ac7c8569dd53b48e1ec4ed216df0dfeeea61
Oct 30 21:40:06.613151 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:06.613059    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jcp77" event={"ID":"10f05285-32c1-4416-82fd-dd2d81a3a8bb","Type":"ContainerStarted","Data":"fac0263d360042f5181075bbd69574d854efc9f8871ed8005ba40c8d7e8d62f3"}
Oct 30 21:40:06.613151 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:06.613143    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jcp77" event={"ID":"10f05285-32c1-4416-82fd-dd2d81a3a8bb","Type":"ContainerStarted","Data":"4d4f29fdcc78c660458f95e32ec83117d571865ad5b45af11033d4f02f56f676"}
Oct 30 21:40:06.615482 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:06.615436    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerStarted","Data":"96eb309f66bddfb6092e1fc26d14ac7c8569dd53b48e1ec4ed216df0dfeeea61"}
Oct 30 21:40:06.648381 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:06.648317    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jcp77" podStartSLOduration=2.54584602 podStartE2EDuration="3.648300013s" totalImagesPullingTime="1.011107774s" totalInitContainerRuntime="91.346219ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:06.646491905 +0000 UTC m=+25.304588123" watchObservedRunningTime="2026-10-30 21:40:06.648300013 +0000 UTC m=+25.306396237"
Oct 30 21:40:07.621319 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:07.621287    4282 generic.go:356] "Generic (PLEG): container finished" podID="36542c81-5234-4762-805e-29d56166bd2b" containerID="37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865" exitCode=0
Oct 30 21:40:07.621788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:07.621379    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerDied","Data":"37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865"}
Oct 30 21:40:07.960782 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:07.960758    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-69cb58c6d7-crnlg"]
Oct 30 21:40:07.991081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:07.991052    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69cb58c6d7-crnlg"]
Oct 30 21:40:07.991258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:07.991188    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.002733 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.002698    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f4ndj\""
Oct 30 21:40:08.004679 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.004657    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:40:08.004959 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.004940    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:40:08.005054 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.004975    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:40:08.007276 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.007256    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:40:08.107686 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.107654    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:40:08.107686 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.107701    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:40:08.135151 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.135127    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-oauth-serving-cert\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.135299 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.135159    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-trusted-ca-bundle\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.135299 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.135182    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-oauth-config\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.135299 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.135246    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b0dfc7-6e8d-4d09-baad-11d336232d06-tmp\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.135428 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.135354    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-config\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.135428 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.135378    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-serving-cert\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.135523 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.135443    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6v78\" (UniqueName: \"kubernetes.io/projected/b8b0dfc7-6e8d-4d09-baad-11d336232d06-kube-api-access-f6v78\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.135570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.135529    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-service-ca\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.236832 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.236753    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-config\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.236832 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.236793    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-serving-cert\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237048 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.236840    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6v78\" (UniqueName: \"kubernetes.io/projected/b8b0dfc7-6e8d-4d09-baad-11d336232d06-kube-api-access-f6v78\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237048 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.236879    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-service-ca\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237048 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.236944    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-oauth-serving-cert\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237048 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.236995    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-trusted-ca-bundle\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237048 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.237027    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-oauth-config\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237288 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.237065    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b0dfc7-6e8d-4d09-baad-11d336232d06-tmp\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237460 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.237443    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b0dfc7-6e8d-4d09-baad-11d336232d06-tmp\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237613 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.237588    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-config\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.237823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.237800    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-oauth-serving-cert\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.238406 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.238378    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-service-ca\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.238542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.238523    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-trusted-ca-bundle\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.239977 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.239954    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-serving-cert\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.240858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.240838    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-oauth-config\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.250515 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.250487    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6v78\" (UniqueName: \"kubernetes.io/projected/b8b0dfc7-6e8d-4d09-baad-11d336232d06-kube-api-access-f6v78\") pod \"console-69cb58c6d7-crnlg\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") " pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.321581 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.321359    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:08.650883 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:08.650798    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69cb58c6d7-crnlg"]
Oct 30 21:40:08.654796 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:40:08.654768    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b0dfc7_6e8d_4d09_baad_11d336232d06.slice/crio-0b2136063e2711e3e44f39fe6212579876d77c7fcb34f1437ec415296cabeb4e WatchSource:0}: Error finding container 0b2136063e2711e3e44f39fe6212579876d77c7fcb34f1437ec415296cabeb4e: Status 404 returned error can't find the container with id 0b2136063e2711e3e44f39fe6212579876d77c7fcb34f1437ec415296cabeb4e
Oct 30 21:40:09.633888 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:09.633855    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-vp2lj" event={"ID":"51815808-5f76-4970-a434-88c3e11b2348","Type":"ContainerStarted","Data":"1ddd98cfe10b0d7040d954283f933d2c196b83295ab601d4b6e4479e78168b7f"}
Oct 30 21:40:09.634010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:09.633987    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:40:09.636807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:09.636786    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerStarted","Data":"6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa"}
Oct 30 21:40:09.636908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:09.636868    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerStarted","Data":"41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164"}
Oct 30 21:40:09.638526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:09.638506    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cb58c6d7-crnlg" event={"ID":"b8b0dfc7-6e8d-4d09-baad-11d336232d06","Type":"ContainerStarted","Data":"0b2136063e2711e3e44f39fe6212579876d77c7fcb34f1437ec415296cabeb4e"}
Oct 30 21:40:09.648982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:09.648961    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-vp2lj"
Oct 30 21:40:09.654671 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:09.654623    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-vp2lj" podStartSLOduration=2.020631527 podStartE2EDuration="19.654609304s" totalImagesPullingTime="17.633977777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:09.652856543 +0000 UTC m=+28.310952767" watchObservedRunningTime="2026-10-30 21:40:09.654609304 +0000 UTC m=+28.312705528"
Oct 30 21:40:09.770424 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:09.768856    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kxgqs"
Oct 30 21:40:10.648258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:10.648097    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerStarted","Data":"6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f"}
Oct 30 21:40:10.648258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:10.648180    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerStarted","Data":"1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b"}
Oct 30 21:40:10.648258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:10.648212    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerStarted","Data":"3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75"}
Oct 30 21:40:11.656443 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:11.656320    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerStarted","Data":"62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a"}
Oct 30 21:40:11.703869 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:11.703814    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.505127864 podStartE2EDuration="6.703799307s" totalImagesPullingTime="4.18265896s" totalInitContainerRuntime="16.012483ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:11.702902369 +0000 UTC m=+30.360998593" watchObservedRunningTime="2026-10-30 21:40:11.703799307 +0000 UTC m=+30.361895529"
Oct 30 21:40:13.665986 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:13.665949    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cb58c6d7-crnlg" event={"ID":"b8b0dfc7-6e8d-4d09-baad-11d336232d06","Type":"ContainerStarted","Data":"86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24"}
Oct 30 21:40:13.686676 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:13.686627    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-69cb58c6d7-crnlg" podStartSLOduration=2.226061011 podStartE2EDuration="6.686613238s" totalImagesPullingTime="4.460552227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:13.686306842 +0000 UTC m=+32.344403069" watchObservedRunningTime="2026-10-30 21:40:13.686613238 +0000 UTC m=+32.344709461"
Oct 30 21:40:15.014167 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:15.013277    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6886d97769-h7tkk"
Oct 30 21:40:16.748385 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.748351    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69cb58c6d7-crnlg"]
Oct 30 21:40:16.784329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.784290    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86bbff4689-9f62t"]
Oct 30 21:40:16.811903 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.811870    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86bbff4689-9f62t"]
Oct 30 21:40:16.812076 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.811996    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:16.920834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.920799    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-service-ca\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:16.920834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.920837    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-oauth-config\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:16.921075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.920857    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-oauth-serving-cert\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:16.921075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.920940    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b1aec7-67a7-4c49-a08e-24f8b186bf46-tmp\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:16.921075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.921004    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-config\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:16.921075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.921040    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-serving-cert\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:16.921292 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.921125    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-trusted-ca-bundle\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:16.921292 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:16.921246    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvk55\" (UniqueName: \"kubernetes.io/projected/27b1aec7-67a7-4c49-a08e-24f8b186bf46-kube-api-access-tvk55\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.022404 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022326    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-service-ca\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.022404 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022365    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-oauth-config\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.022404 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022388    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-oauth-serving-cert\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.022755 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022413    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b1aec7-67a7-4c49-a08e-24f8b186bf46-tmp\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.022755 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022442    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-config\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.022755 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022488    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-serving-cert\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.022755 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022541    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-trusted-ca-bundle\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.022755 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022598    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvk55\" (UniqueName: \"kubernetes.io/projected/27b1aec7-67a7-4c49-a08e-24f8b186bf46-kube-api-access-tvk55\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.023064 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.022878    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b1aec7-67a7-4c49-a08e-24f8b186bf46-tmp\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.023199 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.023143    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-oauth-serving-cert\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.023306 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.023272    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-config\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.023603 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.023576    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-service-ca\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.023697 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.023643    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-trusted-ca-bundle\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.025366 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.025344    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-oauth-config\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.025890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.025869    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-serving-cert\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.036728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.036711    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvk55\" (UniqueName: \"kubernetes.io/projected/27b1aec7-67a7-4c49-a08e-24f8b186bf46-kube-api-access-tvk55\") pod \"console-86bbff4689-9f62t\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") " pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.142449 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.142414    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:17.455190 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.455142    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86bbff4689-9f62t"]
Oct 30 21:40:17.462070 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:40:17.462036    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b1aec7_67a7_4c49_a08e_24f8b186bf46.slice/crio-dbb5b0a54c7dc0b0dad496c16faf1d242c84248852d4208346fecd6306855c42 WatchSource:0}: Error finding container dbb5b0a54c7dc0b0dad496c16faf1d242c84248852d4208346fecd6306855c42: Status 404 returned error can't find the container with id dbb5b0a54c7dc0b0dad496c16faf1d242c84248852d4208346fecd6306855c42
Oct 30 21:40:17.699944 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.699906    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86bbff4689-9f62t" event={"ID":"27b1aec7-67a7-4c49-a08e-24f8b186bf46","Type":"ContainerStarted","Data":"a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507"}
Oct 30 21:40:17.700182 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.699973    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86bbff4689-9f62t" event={"ID":"27b1aec7-67a7-4c49-a08e-24f8b186bf46","Type":"ContainerStarted","Data":"dbb5b0a54c7dc0b0dad496c16faf1d242c84248852d4208346fecd6306855c42"}
Oct 30 21:40:17.719159 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:17.719070    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86bbff4689-9f62t" podStartSLOduration=1.719057855 podStartE2EDuration="1.719057855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:17.718132152 +0000 UTC m=+36.376228386" watchObservedRunningTime="2026-10-30 21:40:17.719057855 +0000 UTC m=+36.377154077"
Oct 30 21:40:18.322257 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:18.322221    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:18.600911 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:18.600845    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8k7kg"
Oct 30 21:40:27.143534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:27.143416    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:27.143534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:27.143509    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:27.148078 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:27.148057    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:27.737286 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:27.737258    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:40:34.648091 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:34.648049    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:40:41.772785 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:41.772724    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-69cb58c6d7-crnlg" podUID="b8b0dfc7-6e8d-4d09-baad-11d336232d06" containerName="console" containerID="cri-o://86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24" gracePeriod=15
Oct 30 21:40:42.046184 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.046114    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:42.213080 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213045    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b0dfc7-6e8d-4d09-baad-11d336232d06-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b0dfc7-6e8d-4d09-baad-11d336232d06-tmp\") pod \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") "
Oct 30 21:40:42.213266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213092    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-oauth-serving-cert\") pod \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") "
Oct 30 21:40:42.213266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213121    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-config\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-config\") pod \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") "
Oct 30 21:40:42.213266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213175    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-trusted-ca-bundle\") pod \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") "
Oct 30 21:40:42.213266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213206    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-serving-cert\") pod \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") "
Oct 30 21:40:42.213266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213260    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b0dfc7-6e8d-4d09-baad-11d336232d06-kube-api-access-f6v78\" (UniqueName: \"kubernetes.io/projected/b8b0dfc7-6e8d-4d09-baad-11d336232d06-kube-api-access-f6v78\") pod \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") "
Oct 30 21:40:42.213586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213302    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-service-ca\") pod \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") "
Oct 30 21:40:42.213586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213331    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-oauth-config\") pod \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\" (UID: \"b8b0dfc7-6e8d-4d09-baad-11d336232d06\") "
Oct 30 21:40:42.213586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213372    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b0dfc7-6e8d-4d09-baad-11d336232d06-tmp" pod "b8b0dfc7-6e8d-4d09-baad-11d336232d06" (UID: "b8b0dfc7-6e8d-4d09-baad-11d336232d06"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:40:42.213586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213531    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-oauth-serving-cert" pod "b8b0dfc7-6e8d-4d09-baad-11d336232d06" (UID: "b8b0dfc7-6e8d-4d09-baad-11d336232d06"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:42.213586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213549    4282 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b0dfc7-6e8d-4d09-baad-11d336232d06-tmp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:42.213586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213571    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-config" pod "b8b0dfc7-6e8d-4d09-baad-11d336232d06" (UID: "b8b0dfc7-6e8d-4d09-baad-11d336232d06"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:42.213860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213640    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-trusted-ca-bundle" pod "b8b0dfc7-6e8d-4d09-baad-11d336232d06" (UID: "b8b0dfc7-6e8d-4d09-baad-11d336232d06"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:42.214020 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.213999    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-service-ca" pod "b8b0dfc7-6e8d-4d09-baad-11d336232d06" (UID: "b8b0dfc7-6e8d-4d09-baad-11d336232d06"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:42.215786 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.215758    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-serving-cert" pod "b8b0dfc7-6e8d-4d09-baad-11d336232d06" (UID: "b8b0dfc7-6e8d-4d09-baad-11d336232d06"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:42.215882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.215794    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-oauth-config" pod "b8b0dfc7-6e8d-4d09-baad-11d336232d06" (UID: "b8b0dfc7-6e8d-4d09-baad-11d336232d06"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:42.215882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.215822    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b0dfc7-6e8d-4d09-baad-11d336232d06-kube-api-access-f6v78" pod "b8b0dfc7-6e8d-4d09-baad-11d336232d06" (UID: "b8b0dfc7-6e8d-4d09-baad-11d336232d06"). InnerVolumeSpecName "kube-api-access-f6v78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:42.314704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.314642    4282 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-trusted-ca-bundle\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:42.314704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.314666    4282 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-serving-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:42.314704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.314675    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6v78\" (UniqueName: \"kubernetes.io/projected/b8b0dfc7-6e8d-4d09-baad-11d336232d06-kube-api-access-f6v78\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:42.314704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.314685    4282 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:42.314704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.314693    4282 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-oauth-config\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:42.314704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.314701    4282 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-oauth-serving-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:42.314704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.314709    4282 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8b0dfc7-6e8d-4d09-baad-11d336232d06-console-config\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:42.783427 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.783397    4282 generic.go:356] "Generic (PLEG): container finished" podID="b8b0dfc7-6e8d-4d09-baad-11d336232d06" containerID="86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24" exitCode=0
Oct 30 21:40:42.783877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.783436    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cb58c6d7-crnlg" event={"ID":"b8b0dfc7-6e8d-4d09-baad-11d336232d06","Type":"ContainerDied","Data":"86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24"}
Oct 30 21:40:42.783877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.783451    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69cb58c6d7-crnlg"
Oct 30 21:40:42.783877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.783495    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69cb58c6d7-crnlg" event={"ID":"b8b0dfc7-6e8d-4d09-baad-11d336232d06","Type":"ContainerDied","Data":"0b2136063e2711e3e44f39fe6212579876d77c7fcb34f1437ec415296cabeb4e"}
Oct 30 21:40:42.783877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.783515    4282 scope.go:122] "RemoveContainer" containerID="86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24"
Oct 30 21:40:42.791903 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.791884    4282 scope.go:122] "RemoveContainer" containerID="86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24"
Oct 30 21:40:42.792189 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:42.792172    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24\": container with ID starting with 86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24 not found: ID does not exist" containerID="86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24"
Oct 30 21:40:42.792241 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.792198    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24"} err="failed to get container status \"86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24\": rpc error: code = NotFound desc = could not find container \"86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24\": container with ID starting with 86562bf19c3e87f307d310be458bfa89ebef9023d02c9d618e8009c7638f1d24 not found: ID does not exist"
Oct 30 21:40:42.803594 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.803572    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69cb58c6d7-crnlg"]
Oct 30 21:40:42.812644 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:42.812621    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69cb58c6d7-crnlg"]
Oct 30 21:40:43.461551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:43.461517    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b0dfc7-6e8d-4d09-baad-11d336232d06" path="/var/lib/kubelet/pods/b8b0dfc7-6e8d-4d09-baad-11d336232d06/volumes"
Oct 30 21:40:43.847805 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:43.847728    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:43.848187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:43.848029    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="alertmanager" containerID="cri-o://41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164" gracePeriod=120
Oct 30 21:40:43.848187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:43.848075    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy-metric" containerID="cri-o://6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f" gracePeriod=120
Oct 30 21:40:43.848187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:43.848123    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="config-reloader" containerID="cri-o://6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa" gracePeriod=120
Oct 30 21:40:43.848187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:43.848124    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy-web" containerID="cri-o://3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75" gracePeriod=120
Oct 30 21:40:43.848187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:43.848167    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy" containerID="cri-o://1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b" gracePeriod=120
Oct 30 21:40:43.848441 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:43.848245    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="prom-label-proxy" containerID="cri-o://62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a" gracePeriod=120
Oct 30 21:40:44.794275 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794241    4282 generic.go:356] "Generic (PLEG): container finished" podID="36542c81-5234-4762-805e-29d56166bd2b" containerID="62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a" exitCode=0
Oct 30 21:40:44.794275 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794266    4282 generic.go:356] "Generic (PLEG): container finished" podID="36542c81-5234-4762-805e-29d56166bd2b" containerID="6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f" exitCode=0
Oct 30 21:40:44.794275 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794272    4282 generic.go:356] "Generic (PLEG): container finished" podID="36542c81-5234-4762-805e-29d56166bd2b" containerID="1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b" exitCode=0
Oct 30 21:40:44.794275 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794278    4282 generic.go:356] "Generic (PLEG): container finished" podID="36542c81-5234-4762-805e-29d56166bd2b" containerID="6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa" exitCode=0
Oct 30 21:40:44.794570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794286    4282 generic.go:356] "Generic (PLEG): container finished" podID="36542c81-5234-4762-805e-29d56166bd2b" containerID="41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164" exitCode=0
Oct 30 21:40:44.794570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794316    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerDied","Data":"62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a"}
Oct 30 21:40:44.794570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794378    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerDied","Data":"6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f"}
Oct 30 21:40:44.794570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794404    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerDied","Data":"1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b"}
Oct 30 21:40:44.794570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794429    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerDied","Data":"6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa"}
Oct 30 21:40:44.794570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:44.794485    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerDied","Data":"41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164"}
Oct 30 21:40:45.131980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.131952    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.135511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135481    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-trusted-ca-bundle\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.135748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135540    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-kube-api-access-t4x5w\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-kube-api-access-t4x5w\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.135748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135579    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-config-volume\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-config-volume\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.135748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135598    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.135748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135618    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-web-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-web-config\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.135748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135654    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-config-out\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.135748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135695    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-main-tls\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.135748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135726    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.136148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135777    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-main-db\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.136148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135803    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-tls-assets\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-tls-assets\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.136148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135801    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-trusted-ca-bundle" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:45.136148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135837    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-metrics-client-ca\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.136148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.135870    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.136148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.136062    4282 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.136757 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.136729    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-main-db" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:40:45.137245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.137214    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-metrics-client-ca" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:40:45.138666 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.138625    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:45.138983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.138946    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-tls-assets" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:45.139145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.139123    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-kube-api-access-t4x5w" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "kube-api-access-t4x5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:40:45.139145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.139126    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-web" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:45.140139 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.140119    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-config-out" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:40:45.140208 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.140136    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-main-tls" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:45.140602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.140575    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-config-volume" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:45.140803 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.140769    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-metric" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:45.149522 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.149494    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-web-config" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:45.237157 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237125    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-cluster-tls-config\") pod \"36542c81-5234-4762-805e-29d56166bd2b\" (UID: \"36542c81-5234-4762-805e-29d56166bd2b\") "
Oct 30 21:40:45.237353 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237338    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4x5w\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-kube-api-access-t4x5w\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237437 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237360    4282 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237437 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237377    4282 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237437 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237391    4282 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-web-config\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237437 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237404    4282 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-config-out\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237437 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237418    4282 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-main-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237664 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237435    4282 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237664 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237452    4282 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/36542c81-5234-4762-805e-29d56166bd2b-alertmanager-main-db\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237664 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237492    4282 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/36542c81-5234-4762-805e-29d56166bd2b-tls-assets\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237664 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237507    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/36542c81-5234-4762-805e-29d56166bd2b-metrics-client-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.237664 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.237523    4282 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.242409 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.242380    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-cluster-tls-config" pod "36542c81-5234-4762-805e-29d56166bd2b" (UID: "36542c81-5234-4762-805e-29d56166bd2b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:40:45.338750 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.338679    4282 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/36542c81-5234-4762-805e-29d56166bd2b-cluster-tls-config\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:40:45.800787 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.800753    4282 generic.go:356] "Generic (PLEG): container finished" podID="36542c81-5234-4762-805e-29d56166bd2b" containerID="3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75" exitCode=0
Oct 30 21:40:45.800953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.800801    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerDied","Data":"3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75"}
Oct 30 21:40:45.800953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.800843    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.800953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.800878    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"36542c81-5234-4762-805e-29d56166bd2b","Type":"ContainerDied","Data":"96eb309f66bddfb6092e1fc26d14ac7c8569dd53b48e1ec4ed216df0dfeeea61"}
Oct 30 21:40:45.800953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.800912    4282 scope.go:122] "RemoveContainer" containerID="62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a"
Oct 30 21:40:45.808638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.808622    4282 scope.go:122] "RemoveContainer" containerID="6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f"
Oct 30 21:40:45.815672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.815649    4282 scope.go:122] "RemoveContainer" containerID="1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b"
Oct 30 21:40:45.820641 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.820622    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:45.822704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.822687    4282 scope.go:122] "RemoveContainer" containerID="3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75"
Oct 30 21:40:45.825091 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.825069    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:45.829715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.829700    4282 scope.go:122] "RemoveContainer" containerID="6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa"
Oct 30 21:40:45.836138 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.836119    4282 scope.go:122] "RemoveContainer" containerID="41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164"
Oct 30 21:40:45.842803 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.842788    4282 scope.go:122] "RemoveContainer" containerID="37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865"
Oct 30 21:40:45.849216 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.849200    4282 scope.go:122] "RemoveContainer" containerID="62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a"
Oct 30 21:40:45.849495 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:45.849457    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a\": container with ID starting with 62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a not found: ID does not exist" containerID="62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a"
Oct 30 21:40:45.849585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.849507    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a"} err="failed to get container status \"62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a\": rpc error: code = NotFound desc = could not find container \"62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a\": container with ID starting with 62767c670538c5df148cd7fd236bf8d661fcfb32d189e40b6ceac0cce2b24b6a not found: ID does not exist"
Oct 30 21:40:45.849585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.849530    4282 scope.go:122] "RemoveContainer" containerID="6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f"
Oct 30 21:40:45.849795 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:45.849776    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f\": container with ID starting with 6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f not found: ID does not exist" containerID="6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f"
Oct 30 21:40:45.849834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.849802    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f"} err="failed to get container status \"6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f\": rpc error: code = NotFound desc = could not find container \"6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f\": container with ID starting with 6106bb26406aeeb5b1569579bd80a3f2916772fdf42c9617c717f461a6748b3f not found: ID does not exist"
Oct 30 21:40:45.849834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.849817    4282 scope.go:122] "RemoveContainer" containerID="1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b"
Oct 30 21:40:45.850044 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:45.850028    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b\": container with ID starting with 1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b not found: ID does not exist" containerID="1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b"
Oct 30 21:40:45.850125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.850051    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b"} err="failed to get container status \"1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b\": rpc error: code = NotFound desc = could not find container \"1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b\": container with ID starting with 1c2614a9f382f2b424f128fbf63ad71c6d5454426aa45853a6516f05a45c240b not found: ID does not exist"
Oct 30 21:40:45.850125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.850070    4282 scope.go:122] "RemoveContainer" containerID="3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75"
Oct 30 21:40:45.850316 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:45.850298    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75\": container with ID starting with 3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75 not found: ID does not exist" containerID="3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75"
Oct 30 21:40:45.850362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.850321    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75"} err="failed to get container status \"3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75\": rpc error: code = NotFound desc = could not find container \"3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75\": container with ID starting with 3d641f085784948c73ead68052a31d3539c3d1375bbf369ec8e759a0a92d1b75 not found: ID does not exist"
Oct 30 21:40:45.850362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.850336    4282 scope.go:122] "RemoveContainer" containerID="6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa"
Oct 30 21:40:45.850595 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:45.850578    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa\": container with ID starting with 6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa not found: ID does not exist" containerID="6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa"
Oct 30 21:40:45.850670 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.850602    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa"} err="failed to get container status \"6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa\": rpc error: code = NotFound desc = could not find container \"6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa\": container with ID starting with 6423d14f179bdebcc915d7f884ff44287fbe15689cf2f16c75974131b73a2daa not found: ID does not exist"
Oct 30 21:40:45.850670 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.850620    4282 scope.go:122] "RemoveContainer" containerID="41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164"
Oct 30 21:40:45.850828 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:45.850811    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164\": container with ID starting with 41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164 not found: ID does not exist" containerID="41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164"
Oct 30 21:40:45.850864 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.850833    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164"} err="failed to get container status \"41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164\": rpc error: code = NotFound desc = could not find container \"41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164\": container with ID starting with 41fe79d0aaaf7a0c128537b489d4dcc7dba760beb9cd59ce8345a4a41982c164 not found: ID does not exist"
Oct 30 21:40:45.850864 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.850847    4282 scope.go:122] "RemoveContainer" containerID="37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865"
Oct 30 21:40:45.851032 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:40:45.851016    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865\": container with ID starting with 37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865 not found: ID does not exist" containerID="37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865"
Oct 30 21:40:45.851081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.851038    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865"} err="failed to get container status \"37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865\": rpc error: code = NotFound desc = could not find container \"37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865\": container with ID starting with 37631921654d6ee7dd98ae07f5d0412f0978aa1eae88b615633d25e5a9f6f865 not found: ID does not exist"
Oct 30 21:40:45.858836 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.858814    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:45.859089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859075    4282 cpu_manager.go:414] "containerMap: removing container" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="alertmanager"
Oct 30 21:40:45.859154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859092    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="alertmanager"
Oct 30 21:40:45.859154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859104    4282 cpu_manager.go:414] "containerMap: removing container" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy"
Oct 30 21:40:45.859154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859111    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy"
Oct 30 21:40:45.859154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859120    4282 cpu_manager.go:414] "containerMap: removing container" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy-metric"
Oct 30 21:40:45.859154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859127    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy-metric"
Oct 30 21:40:45.859154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859141    4282 cpu_manager.go:414] "containerMap: removing container" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="config-reloader"
Oct 30 21:40:45.859154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859148    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="config-reloader"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859157    4282 cpu_manager.go:414] "containerMap: removing container" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy-web"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859163    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy-web"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859172    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b8b0dfc7-6e8d-4d09-baad-11d336232d06" containerName="console"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859179    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b0dfc7-6e8d-4d09-baad-11d336232d06" containerName="console"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859194    4282 cpu_manager.go:414] "containerMap: removing container" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="init-config-reloader"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859201    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="init-config-reloader"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859211    4282 cpu_manager.go:414] "containerMap: removing container" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="prom-label-proxy"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859218    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="prom-label-proxy"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859294    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="alertmanager"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859304    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859313    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy-metric"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859322    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b0dfc7-6e8d-4d09-baad-11d336232d06" containerName="console"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859331    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="config-reloader"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859341    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="kube-rbac-proxy-web"
Oct 30 21:40:45.859540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.859351    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="36542c81-5234-4762-805e-29d56166bd2b" containerName="prom-label-proxy"
Oct 30 21:40:45.884961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.884935    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:45.885053 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.885032    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.888702    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.888706    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.888729    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.888780    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.888876    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.888919    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g8tpb\""
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.889004    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.889014    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:40:45.889504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.889007    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:40:45.889998 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.889754    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:40:45.942404 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942376    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dab523b5-a6c7-4ae1-811d-15509ccf868b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942407    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942429    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dab523b5-a6c7-4ae1-811d-15509ccf868b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942446    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab523b5-a6c7-4ae1-811d-15509ccf868b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942541    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942565    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfhw\" (UniqueName: \"kubernetes.io/projected/dab523b5-a6c7-4ae1-811d-15509ccf868b-kube-api-access-8qfhw\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942669    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab523b5-a6c7-4ae1-811d-15509ccf868b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942717    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-web-config\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942749    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dab523b5-a6c7-4ae1-811d-15509ccf868b-config-out\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942965 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942781    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942965 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942828    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-config-volume\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942965 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942863    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:45.942965 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:45.942890    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043400 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043368    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dab523b5-a6c7-4ae1-811d-15509ccf868b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043400 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043399    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043422    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dab523b5-a6c7-4ae1-811d-15509ccf868b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043444    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab523b5-a6c7-4ae1-811d-15509ccf868b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043489    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043512    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfhw\" (UniqueName: \"kubernetes.io/projected/dab523b5-a6c7-4ae1-811d-15509ccf868b-kube-api-access-8qfhw\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043542    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab523b5-a6c7-4ae1-811d-15509ccf868b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043578    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-web-config\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043604    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dab523b5-a6c7-4ae1-811d-15509ccf868b-config-out\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043631    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043943 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043653    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-config-volume\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043943 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043683    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.043943 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043704    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.044092 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.043936    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dab523b5-a6c7-4ae1-811d-15509ccf868b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.044646 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.044608    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dab523b5-a6c7-4ae1-811d-15509ccf868b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.044920 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.044894    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dab523b5-a6c7-4ae1-811d-15509ccf868b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.047044 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.046710    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.047044 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.047017    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dab523b5-a6c7-4ae1-811d-15509ccf868b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.047044 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.047017    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-web-config\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.047201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.047111    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.047364 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.047344    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dab523b5-a6c7-4ae1-811d-15509ccf868b-config-out\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.047800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.047781    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.047990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.047960    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-config-volume\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.048450 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.048425    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.048708 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.048691    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dab523b5-a6c7-4ae1-811d-15509ccf868b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.055301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.055261    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfhw\" (UniqueName: \"kubernetes.io/projected/dab523b5-a6c7-4ae1-811d-15509ccf868b-kube-api-access-8qfhw\") pod \"alertmanager-main-0\" (UID: \"dab523b5-a6c7-4ae1-811d-15509ccf868b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.346163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.346085    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:40:46.661237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.661163    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:40:46.664643 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:40:46.664609    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab523b5_a6c7_4ae1_811d_15509ccf868b.slice/crio-bb1c0742aa8584db20b181255db2bba0ae35b5bf4e0168e5cfa2380c3a72969f WatchSource:0}: Error finding container bb1c0742aa8584db20b181255db2bba0ae35b5bf4e0168e5cfa2380c3a72969f: Status 404 returned error can't find the container with id bb1c0742aa8584db20b181255db2bba0ae35b5bf4e0168e5cfa2380c3a72969f
Oct 30 21:40:46.806360 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.806325    4282 generic.go:356] "Generic (PLEG): container finished" podID="dab523b5-a6c7-4ae1-811d-15509ccf868b" containerID="3172c1c404e618e89ccfe86d0ce069214ba99fa4f7e63fab9c323bb6d1f65219" exitCode=0
Oct 30 21:40:46.806526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.806419    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dab523b5-a6c7-4ae1-811d-15509ccf868b","Type":"ContainerDied","Data":"3172c1c404e618e89ccfe86d0ce069214ba99fa4f7e63fab9c323bb6d1f65219"}
Oct 30 21:40:46.806526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:46.806520    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dab523b5-a6c7-4ae1-811d-15509ccf868b","Type":"ContainerStarted","Data":"bb1c0742aa8584db20b181255db2bba0ae35b5bf4e0168e5cfa2380c3a72969f"}
Oct 30 21:40:47.460238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.460162    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36542c81-5234-4762-805e-29d56166bd2b" path="/var/lib/kubelet/pods/36542c81-5234-4762-805e-29d56166bd2b/volumes"
Oct 30 21:40:47.814343 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.814269    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dab523b5-a6c7-4ae1-811d-15509ccf868b","Type":"ContainerStarted","Data":"b43a56ae2b3709615b860c37ea5a564ff645b6d9fb8e65a48af2c751b6052e42"}
Oct 30 21:40:47.814343 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.814332    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dab523b5-a6c7-4ae1-811d-15509ccf868b","Type":"ContainerStarted","Data":"c15d65288727fa64df63d9558abe3ece7b77dd7cc55724430450368e8ae20fea"}
Oct 30 21:40:47.814559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.814363    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dab523b5-a6c7-4ae1-811d-15509ccf868b","Type":"ContainerStarted","Data":"d23959528ac57118a3a506191aa791d8440f9762e4a7c1c00f2976d07c55cd35"}
Oct 30 21:40:47.814559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.814388    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dab523b5-a6c7-4ae1-811d-15509ccf868b","Type":"ContainerStarted","Data":"663a31d5ecd146f7fdbcb28512622e32cadc64691c542149ce67bd5e205fa557"}
Oct 30 21:40:47.814559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.814411    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dab523b5-a6c7-4ae1-811d-15509ccf868b","Type":"ContainerStarted","Data":"9902038b2b80351c11cb977ed49471378aa36d2081f1f95566f77eccad718b46"}
Oct 30 21:40:47.814559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.814440    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dab523b5-a6c7-4ae1-811d-15509ccf868b","Type":"ContainerStarted","Data":"04e4f4e93548e437cdac781ca75edb4ade21609b28ef858c092fbf1d48d1cd4d"}
Oct 30 21:40:47.844858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.844787    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8296017620000002 podStartE2EDuration="2.844772706s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.170944ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:47.843337521 +0000 UTC m=+66.501433745" watchObservedRunningTime="2026-10-30 21:40:47.844772706 +0000 UTC m=+66.502868928"
Oct 30 21:40:47.879929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.879901    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-546f884c57-bf88n"]
Oct 30 21:40:47.905153 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.905120    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-546f884c57-bf88n"]
Oct 30 21:40:47.905302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.905266    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.909716 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.909693    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 21:40:47.910923 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.910902    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 21:40:47.960436 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.960408    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-serving-certs-ca-bundle\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.960600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.960504    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.960600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.960523    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98p9r\" (UniqueName: \"kubernetes.io/projected/551f22c1-58c2-440d-9326-d78f3a23941d-kube-api-access-98p9r\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.960600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.960542    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.960600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.960562    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-secret-telemeter-client\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.960751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.960663    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-telemeter-client-tls\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.960751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.960702    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-metrics-client-ca\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.960751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.960727    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-federate-client-tls\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:47.969105 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.969085    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zvbpl\""
Oct 30 21:40:47.969105 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.969085    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 21:40:47.969234 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.969085    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 21:40:47.969234 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.969095    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 21:40:47.969234 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:47.969096    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 21:40:48.061246 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.061214    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-serving-certs-ca-bundle\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.061402 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.061255    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.061402 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.061270    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98p9r\" (UniqueName: \"kubernetes.io/projected/551f22c1-58c2-440d-9326-d78f3a23941d-kube-api-access-98p9r\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.061402 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.061380    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.061668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.061421    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-secret-telemeter-client\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.061668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.061482    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-telemeter-client-tls\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.061668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.061526    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-metrics-client-ca\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.061668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.061561    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-federate-client-tls\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.062101 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.062040    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-serving-certs-ca-bundle\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.062289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.062256    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-metrics-client-ca\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.062571 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.062333    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/551f22c1-58c2-440d-9326-d78f3a23941d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.064020 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.063996    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.064236 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.064217    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-federate-client-tls\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.064534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.064458    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-telemeter-client-tls\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.064534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.064501    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/551f22c1-58c2-440d-9326-d78f3a23941d-secret-telemeter-client\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.073025 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.073002    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98p9r\" (UniqueName: \"kubernetes.io/projected/551f22c1-58c2-440d-9326-d78f3a23941d-kube-api-access-98p9r\") pod \"telemeter-client-546f884c57-bf88n\" (UID: \"551f22c1-58c2-440d-9326-d78f3a23941d\") " pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.257057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.257026    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-546f884c57-bf88n"
Oct 30 21:40:48.587826 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.587757    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-546f884c57-bf88n"]
Oct 30 21:40:48.594362 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:40:48.594327    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551f22c1_58c2_440d_9326_d78f3a23941d.slice/crio-eb31d80f27a11662e232f5b98910bec37289a0d4d4e50608cddb25cae870ce3e WatchSource:0}: Error finding container eb31d80f27a11662e232f5b98910bec37289a0d4d4e50608cddb25cae870ce3e: Status 404 returned error can't find the container with id eb31d80f27a11662e232f5b98910bec37289a0d4d4e50608cddb25cae870ce3e
Oct 30 21:40:48.819429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:48.819396    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-546f884c57-bf88n" event={"ID":"551f22c1-58c2-440d-9326-d78f3a23941d","Type":"ContainerStarted","Data":"eb31d80f27a11662e232f5b98910bec37289a0d4d4e50608cddb25cae870ce3e"}
Oct 30 21:40:50.832858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:50.832833    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-546f884c57-bf88n" event={"ID":"551f22c1-58c2-440d-9326-d78f3a23941d","Type":"ContainerStarted","Data":"1f586323806161610d009439f84340cd4c11d17426793d123d97319aef5f13ae"}
Oct 30 21:40:50.833138 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:50.832909    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-546f884c57-bf88n" event={"ID":"551f22c1-58c2-440d-9326-d78f3a23941d","Type":"ContainerStarted","Data":"b4da7a09c06856cd1775b4684b5bf4fc6a7b9feb9e2692de5b216d0696285a9b"}
Oct 30 21:40:51.838255 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.838218    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-546f884c57-bf88n" event={"ID":"551f22c1-58c2-440d-9326-d78f3a23941d","Type":"ContainerStarted","Data":"617e199c6a0d23d207dbb20f11ac5d0814d247f4c886e17a1a927e6dbf76919e"}
Oct 30 21:40:51.862980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.862940    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-546f884c57-bf88n" podStartSLOduration=2.860845391 podStartE2EDuration="4.862926928s" totalImagesPullingTime="2.002081537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:51.861758018 +0000 UTC m=+70.519854241" watchObservedRunningTime="2026-10-30 21:40:51.862926928 +0000 UTC m=+70.521023151"
Oct 30 21:40:51.954533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.954505    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cc5b47456-whgcw"]
Oct 30 21:40:51.971714 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.971672    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cc5b47456-whgcw"]
Oct 30 21:40:51.971842 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.971790    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:51.992931 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.992907    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-service-ca\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:51.993047 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.992945    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9dhx\" (UniqueName: \"kubernetes.io/projected/e602b0d2-cffb-48b0-9366-a72116485777-kube-api-access-h9dhx\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:51.993047 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.992964    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-console-config\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:51.993114 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.993060    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-oauth-config\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:51.993114 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.993091    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e602b0d2-cffb-48b0-9366-a72116485777-tmp\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:51.993178 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.993117    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-oauth-serving-cert\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:51.993178 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.993140    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-serving-cert\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:51.993266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:51.993246    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-trusted-ca-bundle\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.093904 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.093839    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-service-ca\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.093904 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.093878    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9dhx\" (UniqueName: \"kubernetes.io/projected/e602b0d2-cffb-48b0-9366-a72116485777-kube-api-access-h9dhx\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.093904 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.093897    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-console-config\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094113 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.093935    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-oauth-config\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094113 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.093956    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e602b0d2-cffb-48b0-9366-a72116485777-tmp\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094113 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.093990    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-oauth-serving-cert\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094113 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.094023    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-serving-cert\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094113 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.094054    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-trusted-ca-bundle\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094531 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.094452    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e602b0d2-cffb-48b0-9366-a72116485777-tmp\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094684 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.094660    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-service-ca\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.094742    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-oauth-serving-cert\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.094864    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-trusted-ca-bundle\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.094926 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.094811    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-console-config\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.096561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.096538    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-oauth-config\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.096759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.096741    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-serving-cert\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.105608 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.105587    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9dhx\" (UniqueName: \"kubernetes.io/projected/e602b0d2-cffb-48b0-9366-a72116485777-kube-api-access-h9dhx\") pod \"console-6cc5b47456-whgcw\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") " pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.295692 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.295657    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:40:52.604498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.604430    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cc5b47456-whgcw"]
Oct 30 21:40:52.608073 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:40:52.608043    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode602b0d2_cffb_48b0_9366_a72116485777.slice/crio-0bd920b42736d1205bc5202fcdfefb81bef499b071764520f3682ce79a38f3a1 WatchSource:0}: Error finding container 0bd920b42736d1205bc5202fcdfefb81bef499b071764520f3682ce79a38f3a1: Status 404 returned error can't find the container with id 0bd920b42736d1205bc5202fcdfefb81bef499b071764520f3682ce79a38f3a1
Oct 30 21:40:52.847716 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.847674    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc5b47456-whgcw" event={"ID":"e602b0d2-cffb-48b0-9366-a72116485777","Type":"ContainerStarted","Data":"b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0"}
Oct 30 21:40:52.848079 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.847770    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc5b47456-whgcw" event={"ID":"e602b0d2-cffb-48b0-9366-a72116485777","Type":"ContainerStarted","Data":"0bd920b42736d1205bc5202fcdfefb81bef499b071764520f3682ce79a38f3a1"}
Oct 30 21:40:52.869854 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:40:52.869804    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6cc5b47456-whgcw" podStartSLOduration=1.8697881710000002 podStartE2EDuration="1.869788171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:52.868544426 +0000 UTC m=+71.526640628" watchObservedRunningTime="2026-10-30 21:40:52.869788171 +0000 UTC m=+71.527884398"
Oct 30 21:41:02.296371 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:02.296336    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:41:02.296919 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:02.296395    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:41:02.300903 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:02.300880    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:41:02.890421 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:02.890387    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:41:02.938962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:02.938929    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86bbff4689-9f62t"]
Oct 30 21:41:04.647094 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:41:04.647060    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:41:19.940600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:19.940569    4282 generic.go:356] "Generic (PLEG): container finished" podID="2f74abea-1ffe-482b-bc0f-31d297aca1f8" containerID="ccb31c31ffdcd4d11fb190d3eed7309c41d2d397f00acfb46b57f957c8647d82" exitCode=0
Oct 30 21:41:19.940908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:19.940651    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2" event={"ID":"2f74abea-1ffe-482b-bc0f-31d297aca1f8","Type":"ContainerDied","Data":"ccb31c31ffdcd4d11fb190d3eed7309c41d2d397f00acfb46b57f957c8647d82"}
Oct 30 21:41:21.071209 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.071188    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:41:21.214383 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.214303    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-service-ca-bundle\") pod \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") "
Oct 30 21:41:21.214383 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.214362    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f74abea-1ffe-482b-bc0f-31d297aca1f8-kube-api-access-7szr7\" (UniqueName: \"kubernetes.io/projected/2f74abea-1ffe-482b-bc0f-31d297aca1f8-kube-api-access-7szr7\") pod \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") "
Oct 30 21:41:21.214625 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.214388    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f74abea-1ffe-482b-bc0f-31d297aca1f8-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f74abea-1ffe-482b-bc0f-31d297aca1f8-archives-path\") pod \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") "
Oct 30 21:41:21.214625 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.214458    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-trusted-ca-bundle\") pod \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\" (UID: \"2f74abea-1ffe-482b-bc0f-31d297aca1f8\") "
Oct 30 21:41:21.214766 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.214737    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f74abea-1ffe-482b-bc0f-31d297aca1f8-archives-path" pod "2f74abea-1ffe-482b-bc0f-31d297aca1f8" (UID: "2f74abea-1ffe-482b-bc0f-31d297aca1f8"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:21.214823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.214783    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-service-ca-bundle" pod "2f74abea-1ffe-482b-bc0f-31d297aca1f8" (UID: "2f74abea-1ffe-482b-bc0f-31d297aca1f8"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:21.214823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.214805    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-trusted-ca-bundle" pod "2f74abea-1ffe-482b-bc0f-31d297aca1f8" (UID: "2f74abea-1ffe-482b-bc0f-31d297aca1f8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:21.216737 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.216716    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f74abea-1ffe-482b-bc0f-31d297aca1f8-kube-api-access-7szr7" pod "2f74abea-1ffe-482b-bc0f-31d297aca1f8" (UID: "2f74abea-1ffe-482b-bc0f-31d297aca1f8"). InnerVolumeSpecName "kube-api-access-7szr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:21.315709 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.315685    4282 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-service-ca-bundle\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:21.315709 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.315706    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szr7\" (UniqueName: \"kubernetes.io/projected/2f74abea-1ffe-482b-bc0f-31d297aca1f8-kube-api-access-7szr7\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:21.315830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.315716    4282 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2f74abea-1ffe-482b-bc0f-31d297aca1f8-archives-path\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:21.315830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.315725    4282 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f74abea-1ffe-482b-bc0f-31d297aca1f8-trusted-ca-bundle\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:21.948770 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.948737    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2" event={"ID":"2f74abea-1ffe-482b-bc0f-31d297aca1f8","Type":"ContainerDied","Data":"33368d94faa2393d6fd9c9de872c50ecc92b2412e9e5d0e970e2a55e5b43e50c"}
Oct 30 21:41:21.948770 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.948756    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-5m9gv-6wpk2"
Oct 30 21:41:21.948955 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:21.948805    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33368d94faa2393d6fd9c9de872c50ecc92b2412e9e5d0e970e2a55e5b43e50c"
Oct 30 21:41:27.958790 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:27.958755    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86bbff4689-9f62t" podUID="27b1aec7-67a7-4c49-a08e-24f8b186bf46" containerName="console" containerID="cri-o://a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507" gracePeriod=15
Oct 30 21:41:28.200728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.200706    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:41:28.260478 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260394    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-oauth-serving-cert\") pod \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") "
Oct 30 21:41:28.260478 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260434    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27b1aec7-67a7-4c49-a08e-24f8b186bf46-tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b1aec7-67a7-4c49-a08e-24f8b186bf46-tmp\") pod \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") "
Oct 30 21:41:28.260666 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260484    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b1aec7-67a7-4c49-a08e-24f8b186bf46-kube-api-access-tvk55\" (UniqueName: \"kubernetes.io/projected/27b1aec7-67a7-4c49-a08e-24f8b186bf46-kube-api-access-tvk55\") pod \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") "
Oct 30 21:41:28.260666 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260524    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-serving-cert\") pod \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") "
Oct 30 21:41:28.260666 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260561    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-oauth-config\") pod \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") "
Oct 30 21:41:28.260793 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260745    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27b1aec7-67a7-4c49-a08e-24f8b186bf46-tmp" pod "27b1aec7-67a7-4c49-a08e-24f8b186bf46" (UID: "27b1aec7-67a7-4c49-a08e-24f8b186bf46"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:41:28.260856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260792    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-service-ca\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-service-ca\") pod \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") "
Oct 30 21:41:28.260856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260826    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-oauth-serving-cert" pod "27b1aec7-67a7-4c49-a08e-24f8b186bf46" (UID: "27b1aec7-67a7-4c49-a08e-24f8b186bf46"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:28.260856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260834    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-trusted-ca-bundle\") pod \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") "
Oct 30 21:41:28.260995 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.260877    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-config\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-config\") pod \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\" (UID: \"27b1aec7-67a7-4c49-a08e-24f8b186bf46\") "
Oct 30 21:41:28.261127 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.261108    4282 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-oauth-serving-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:28.261243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.261216    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-service-ca" pod "27b1aec7-67a7-4c49-a08e-24f8b186bf46" (UID: "27b1aec7-67a7-4c49-a08e-24f8b186bf46"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:28.261324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.261229    4282 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27b1aec7-67a7-4c49-a08e-24f8b186bf46-tmp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:28.261367 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.261327    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-trusted-ca-bundle" pod "27b1aec7-67a7-4c49-a08e-24f8b186bf46" (UID: "27b1aec7-67a7-4c49-a08e-24f8b186bf46"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:28.261518 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.261492    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-config" pod "27b1aec7-67a7-4c49-a08e-24f8b186bf46" (UID: "27b1aec7-67a7-4c49-a08e-24f8b186bf46"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:41:28.262977 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.262952    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-serving-cert" pod "27b1aec7-67a7-4c49-a08e-24f8b186bf46" (UID: "27b1aec7-67a7-4c49-a08e-24f8b186bf46"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:41:28.263381 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.263363    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b1aec7-67a7-4c49-a08e-24f8b186bf46-kube-api-access-tvk55" pod "27b1aec7-67a7-4c49-a08e-24f8b186bf46" (UID: "27b1aec7-67a7-4c49-a08e-24f8b186bf46"). InnerVolumeSpecName "kube-api-access-tvk55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:28.263423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.263376    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-oauth-config" pod "27b1aec7-67a7-4c49-a08e-24f8b186bf46" (UID: "27b1aec7-67a7-4c49-a08e-24f8b186bf46"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:41:28.362424 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.362391    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvk55\" (UniqueName: \"kubernetes.io/projected/27b1aec7-67a7-4c49-a08e-24f8b186bf46-kube-api-access-tvk55\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:28.362424 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.362420    4282 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-serving-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:28.362602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.362434    4282 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-oauth-config\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:28.362602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.362448    4282 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:28.362602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.362460    4282 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-trusted-ca-bundle\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:28.362602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.362495    4282 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b1aec7-67a7-4c49-a08e-24f8b186bf46-console-config\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:41:28.979563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.979530    4282 generic.go:356] "Generic (PLEG): container finished" podID="27b1aec7-67a7-4c49-a08e-24f8b186bf46" containerID="a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507" exitCode=0
Oct 30 21:41:28.979962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.979574    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86bbff4689-9f62t"
Oct 30 21:41:28.979962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.979617    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86bbff4689-9f62t" event={"ID":"27b1aec7-67a7-4c49-a08e-24f8b186bf46","Type":"ContainerDied","Data":"a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507"}
Oct 30 21:41:28.979962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.979671    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86bbff4689-9f62t" event={"ID":"27b1aec7-67a7-4c49-a08e-24f8b186bf46","Type":"ContainerDied","Data":"dbb5b0a54c7dc0b0dad496c16faf1d242c84248852d4208346fecd6306855c42"}
Oct 30 21:41:28.979962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.979692    4282 scope.go:122] "RemoveContainer" containerID="a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507"
Oct 30 21:41:28.990069 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.990048    4282 scope.go:122] "RemoveContainer" containerID="a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507"
Oct 30 21:41:28.990345 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:41:28.990327    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507\": container with ID starting with a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507 not found: ID does not exist" containerID="a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507"
Oct 30 21:41:28.990407 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:28.990354    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507"} err="failed to get container status \"a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507\": rpc error: code = NotFound desc = could not find container \"a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507\": container with ID starting with a3ec45aa7524b674cf054b117d36e6f981069025d76a5468032d0ac6ca8e5507 not found: ID does not exist"
Oct 30 21:41:29.001916 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:29.001895    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86bbff4689-9f62t"]
Oct 30 21:41:29.006288 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:29.006268    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86bbff4689-9f62t"]
Oct 30 21:41:29.460352 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:29.460317    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27b1aec7-67a7-4c49-a08e-24f8b186bf46" path="/var/lib/kubelet/pods/27b1aec7-67a7-4c49-a08e-24f8b186bf46/volumes"
Oct 30 21:41:34.647358 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:41:34.647318    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:41:59.474596 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.474525    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cbc7cf9f7-vvh2w"]
Oct 30 21:41:59.475021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.474771    4282 cpu_manager.go:414] "containerMap: removing container" podUID="2f74abea-1ffe-482b-bc0f-31d297aca1f8" containerName="insights-gathering"
Oct 30 21:41:59.475021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.474783    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f74abea-1ffe-482b-bc0f-31d297aca1f8" containerName="insights-gathering"
Oct 30 21:41:59.475021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.474797    4282 cpu_manager.go:414] "containerMap: removing container" podUID="27b1aec7-67a7-4c49-a08e-24f8b186bf46" containerName="console"
Oct 30 21:41:59.475021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.474802    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b1aec7-67a7-4c49-a08e-24f8b186bf46" containerName="console"
Oct 30 21:41:59.475021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.474848    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f74abea-1ffe-482b-bc0f-31d297aca1f8" containerName="insights-gathering"
Oct 30 21:41:59.475021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.474856    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b1aec7-67a7-4c49-a08e-24f8b186bf46" containerName="console"
Oct 30 21:41:59.477777 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.477715    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.491232 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.491210    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbc7cf9f7-vvh2w"]
Oct 30 21:41:59.580944 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.580913    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-serving-cert\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.581072 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.580950    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-config\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.581072 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.580974    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-oauth-config\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.581072 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.581045    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-oauth-serving-cert\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.581212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.581084    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95sw\" (UniqueName: \"kubernetes.io/projected/f3baa9af-6804-4f85-a6e4-0661ebfd541f-kube-api-access-f95sw\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.581212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.581114    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-trusted-ca-bundle\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.581212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.581130    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-service-ca\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.581212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.581160    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3baa9af-6804-4f85-a6e4-0661ebfd541f-tmp\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682187    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-oauth-serving-cert\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682215    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95sw\" (UniqueName: \"kubernetes.io/projected/f3baa9af-6804-4f85-a6e4-0661ebfd541f-kube-api-access-f95sw\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682241    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-trusted-ca-bundle\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682255    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-service-ca\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682277    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3baa9af-6804-4f85-a6e4-0661ebfd541f-tmp\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682313    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-serving-cert\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682346    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-config\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682695 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682382    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-oauth-config\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.682794 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.682769    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3baa9af-6804-4f85-a6e4-0661ebfd541f-tmp\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.683240 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.683220    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-config\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.683348 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.683174    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-service-ca\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.683348 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.683253    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-oauth-serving-cert\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.683533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.683512    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3baa9af-6804-4f85-a6e4-0661ebfd541f-trusted-ca-bundle\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.684926 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.684907    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-oauth-config\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.685091 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.685076    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3baa9af-6804-4f85-a6e4-0661ebfd541f-console-serving-cert\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.693846 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.693823    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95sw\" (UniqueName: \"kubernetes.io/projected/f3baa9af-6804-4f85-a6e4-0661ebfd541f-kube-api-access-f95sw\") pod \"console-cbc7cf9f7-vvh2w\" (UID: \"f3baa9af-6804-4f85-a6e4-0661ebfd541f\") " pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:41:59.792981 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:41:59.792891    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:42:00.071634 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:00.071547    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbc7cf9f7-vvh2w"]
Oct 30 21:42:00.076382 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:42:00.076355    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3baa9af_6804_4f85_a6e4_0661ebfd541f.slice/crio-8b14b13740a41aa648e6ef5da5ba606a7cadee68e40f5d62a0853948ac0304b9 WatchSource:0}: Error finding container 8b14b13740a41aa648e6ef5da5ba606a7cadee68e40f5d62a0853948ac0304b9: Status 404 returned error can't find the container with id 8b14b13740a41aa648e6ef5da5ba606a7cadee68e40f5d62a0853948ac0304b9
Oct 30 21:42:00.088998 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:00.088972    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc7cf9f7-vvh2w" event={"ID":"f3baa9af-6804-4f85-a6e4-0661ebfd541f","Type":"ContainerStarted","Data":"8b14b13740a41aa648e6ef5da5ba606a7cadee68e40f5d62a0853948ac0304b9"}
Oct 30 21:42:01.093744 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:01.093708    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbc7cf9f7-vvh2w" event={"ID":"f3baa9af-6804-4f85-a6e4-0661ebfd541f","Type":"ContainerStarted","Data":"cbcbd0663e14b373654957dfa6aed5ab92da75996878eead48de64a764b9d676"}
Oct 30 21:42:01.114654 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:01.114598    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cbc7cf9f7-vvh2w" podStartSLOduration=2.114581214 podStartE2EDuration="2.114581214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:01.113126349 +0000 UTC m=+139.771222569" watchObservedRunningTime="2026-10-30 21:42:01.114581214 +0000 UTC m=+139.772677445"
Oct 30 21:42:04.646068 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:42:04.646025    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:42:09.793642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:09.793600    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:42:09.794145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:09.793660    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:42:09.798362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:09.798339    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:42:10.127958 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:10.127870    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cbc7cf9f7-vvh2w"
Oct 30 21:42:10.182735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:10.182700    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cc5b47456-whgcw"]
Oct 30 21:42:34.646776 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:42:34.646733    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:42:35.202088 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.202050    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6cc5b47456-whgcw" podUID="e602b0d2-cffb-48b0-9366-a72116485777" containerName="console" containerID="cri-o://b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0" gracePeriod=15
Oct 30 21:42:35.448417 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.448392    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:42:35.535066 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.534998    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-oauth-config\") pod \"e602b0d2-cffb-48b0-9366-a72116485777\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") "
Oct 30 21:42:35.535066 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535043    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e602b0d2-cffb-48b0-9366-a72116485777-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e602b0d2-cffb-48b0-9366-a72116485777-tmp\") pod \"e602b0d2-cffb-48b0-9366-a72116485777\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") "
Oct 30 21:42:35.535243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535073    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-oauth-serving-cert\") pod \"e602b0d2-cffb-48b0-9366-a72116485777\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") "
Oct 30 21:42:35.535243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535104    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-service-ca\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-service-ca\") pod \"e602b0d2-cffb-48b0-9366-a72116485777\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") "
Oct 30 21:42:35.535243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535125    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-console-config\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-console-config\") pod \"e602b0d2-cffb-48b0-9366-a72116485777\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") "
Oct 30 21:42:35.535243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535170    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-serving-cert\") pod \"e602b0d2-cffb-48b0-9366-a72116485777\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") "
Oct 30 21:42:35.535243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535198    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-trusted-ca-bundle\") pod \"e602b0d2-cffb-48b0-9366-a72116485777\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") "
Oct 30 21:42:35.535243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535241    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e602b0d2-cffb-48b0-9366-a72116485777-kube-api-access-h9dhx\" (UniqueName: \"kubernetes.io/projected/e602b0d2-cffb-48b0-9366-a72116485777-kube-api-access-h9dhx\") pod \"e602b0d2-cffb-48b0-9366-a72116485777\" (UID: \"e602b0d2-cffb-48b0-9366-a72116485777\") "
Oct 30 21:42:35.535574 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535430    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e602b0d2-cffb-48b0-9366-a72116485777-tmp" pod "e602b0d2-cffb-48b0-9366-a72116485777" (UID: "e602b0d2-cffb-48b0-9366-a72116485777"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:42:35.535827 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535575    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-oauth-serving-cert" pod "e602b0d2-cffb-48b0-9366-a72116485777" (UID: "e602b0d2-cffb-48b0-9366-a72116485777"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:35.536093 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535695    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-console-config" pod "e602b0d2-cffb-48b0-9366-a72116485777" (UID: "e602b0d2-cffb-48b0-9366-a72116485777"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:35.536093 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535784    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-trusted-ca-bundle" pod "e602b0d2-cffb-48b0-9366-a72116485777" (UID: "e602b0d2-cffb-48b0-9366-a72116485777"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:35.536093 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.535795    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-service-ca" pod "e602b0d2-cffb-48b0-9366-a72116485777" (UID: "e602b0d2-cffb-48b0-9366-a72116485777"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:42:35.537328 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.537298    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-oauth-config" pod "e602b0d2-cffb-48b0-9366-a72116485777" (UID: "e602b0d2-cffb-48b0-9366-a72116485777"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:35.537574 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.537546    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-serving-cert" pod "e602b0d2-cffb-48b0-9366-a72116485777" (UID: "e602b0d2-cffb-48b0-9366-a72116485777"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:35.537655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.537596    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e602b0d2-cffb-48b0-9366-a72116485777-kube-api-access-h9dhx" pod "e602b0d2-cffb-48b0-9366-a72116485777" (UID: "e602b0d2-cffb-48b0-9366-a72116485777"). InnerVolumeSpecName "kube-api-access-h9dhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:35.636480 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.636432    4282 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-console-config\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:35.636570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.636486    4282 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-serving-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:35.636570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.636509    4282 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-trusted-ca-bundle\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:35.636570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.636521    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9dhx\" (UniqueName: \"kubernetes.io/projected/e602b0d2-cffb-48b0-9366-a72116485777-kube-api-access-h9dhx\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:35.636570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.636532    4282 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e602b0d2-cffb-48b0-9366-a72116485777-console-oauth-config\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:35.636570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.636541    4282 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e602b0d2-cffb-48b0-9366-a72116485777-tmp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:35.636570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.636549    4282 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-oauth-serving-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:35.636570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:35.636557    4282 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e602b0d2-cffb-48b0-9366-a72116485777-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:42:36.211567 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.211533    4282 generic.go:356] "Generic (PLEG): container finished" podID="e602b0d2-cffb-48b0-9366-a72116485777" containerID="b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0" exitCode=0
Oct 30 21:42:36.211947 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.211582    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cc5b47456-whgcw"
Oct 30 21:42:36.211947 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.211622    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc5b47456-whgcw" event={"ID":"e602b0d2-cffb-48b0-9366-a72116485777","Type":"ContainerDied","Data":"b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0"}
Oct 30 21:42:36.211947 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.211675    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cc5b47456-whgcw" event={"ID":"e602b0d2-cffb-48b0-9366-a72116485777","Type":"ContainerDied","Data":"0bd920b42736d1205bc5202fcdfefb81bef499b071764520f3682ce79a38f3a1"}
Oct 30 21:42:36.211947 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.211695    4282 scope.go:122] "RemoveContainer" containerID="b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0"
Oct 30 21:42:36.219960 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.219945    4282 scope.go:122] "RemoveContainer" containerID="b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0"
Oct 30 21:42:36.220175 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:42:36.220160    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0\": container with ID starting with b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0 not found: ID does not exist" containerID="b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0"
Oct 30 21:42:36.220215 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.220188    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0"} err="failed to get container status \"b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0\": rpc error: code = NotFound desc = could not find container \"b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0\": container with ID starting with b384f1bc12fac69e75240d3f9a78c85c6d84a06038d7bdccb351ce109ecd0ad0 not found: ID does not exist"
Oct 30 21:42:36.232031 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.232010    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cc5b47456-whgcw"]
Oct 30 21:42:36.236429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:36.236412    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cc5b47456-whgcw"]
Oct 30 21:42:37.460930 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:42:37.460899    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e602b0d2-cffb-48b0-9366-a72116485777" path="/var/lib/kubelet/pods/e602b0d2-cffb-48b0-9366-a72116485777/volumes"
Oct 30 21:43:04.647634 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:43:04.647599    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:43:34.647287 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:43:34.647240    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:43:46.765861 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.765828    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"]
Oct 30 21:43:46.766644 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.766623    4282 cpu_manager.go:414] "containerMap: removing container" podUID="e602b0d2-cffb-48b0-9366-a72116485777" containerName="console"
Oct 30 21:43:46.766752 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.766645    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e602b0d2-cffb-48b0-9366-a72116485777" containerName="console"
Oct 30 21:43:46.766752 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.766726    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e602b0d2-cffb-48b0-9366-a72116485777" containerName="console"
Oct 30 21:43:46.769716 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.769697    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.781789 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.781771    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 21:43:46.781789 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.781785    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 21:43:46.781923 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.781798    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-hzztr\""
Oct 30 21:43:46.787879 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.787860    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"]
Oct 30 21:43:46.834009 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.833985    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.834119 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.834017    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f7w\" (UniqueName: \"kubernetes.io/projected/e243a405-5977-4403-ad44-cbfa678a8743-kube-api-access-49f7w\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.834119 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.834037    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.935320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.935293    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.935429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.935324    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f7w\" (UniqueName: \"kubernetes.io/projected/e243a405-5977-4403-ad44-cbfa678a8743-kube-api-access-49f7w\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.935429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.935348    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.935748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.935728    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.935748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.935740    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.957118 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.957088    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f7w\" (UniqueName: \"kubernetes.io/projected/e243a405-5977-4403-ad44-cbfa678a8743-kube-api-access-49f7w\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:46.986278 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.986251    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"]
Oct 30 21:43:46.989500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:46.989486    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.000793 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.000773    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"]
Oct 30 21:43:47.036047 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.035991    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.036047 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.036041    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.036187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.036061    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg9t\" (UniqueName: \"kubernetes.io/projected/915267bd-17be-4c23-a33b-9470d3b48394-kube-api-access-4kg9t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.053413 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.053391    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-mz5df\""
Oct 30 21:43:47.084730 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.084693    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:43:47.136982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.136957    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.137123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.136994    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg9t\" (UniqueName: \"kubernetes.io/projected/915267bd-17be-4c23-a33b-9470d3b48394-kube-api-access-4kg9t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.137123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.137030    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.137337 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.137321    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.137433 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.137409    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.148976 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.148953    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg9t\" (UniqueName: \"kubernetes.io/projected/915267bd-17be-4c23-a33b-9470d3b48394-kube-api-access-4kg9t\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.305353 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.305283    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:43:47.345305 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.345277    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"]
Oct 30 21:43:47.348608 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.348585    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.360329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.359906    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:43:47.360329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.359948    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:43:47.360329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.359974    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 21:43:47.360329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.360069    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:43:47.360329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.360164    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lx85x\""
Oct 30 21:43:47.360329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.360232    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 21:43:47.360329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.360283    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 21:43:47.367802 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.367728    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"]
Oct 30 21:43:47.368778 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.368758    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"]
Oct 30 21:43:47.370661 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:43:47.370633    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode243a405_5977_4403_ad44_cbfa678a8743.slice/crio-4bdb4d681170bb744a697fb2bf430b62c4542650cdca1ea9b6e59c077812ff6b WatchSource:0}: Error finding container 4bdb4d681170bb744a697fb2bf430b62c4542650cdca1ea9b6e59c077812ff6b: Status 404 returned error can't find the container with id 4bdb4d681170bb744a697fb2bf430b62c4542650cdca1ea9b6e59c077812ff6b
Oct 30 21:43:47.439724 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.439688    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.439860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.439733    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.439860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.439801    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a76223c-2b65-4f5c-b7fc-cac2596c8899-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.439860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.439828    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.439860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.439846    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.440011 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.439876    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjr7\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-kube-api-access-jjjr7\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.440011 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.439953    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.451285 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.451259    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl" event={"ID":"e243a405-5977-4403-ad44-cbfa678a8743","Type":"ContainerStarted","Data":"4bdb4d681170bb744a697fb2bf430b62c4542650cdca1ea9b6e59c077812ff6b"}
Oct 30 21:43:47.540938 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.540907    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.541057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.540972    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.541057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.541006    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a76223c-2b65-4f5c-b7fc-cac2596c8899-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.541057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.541030    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.541195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.541056    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.541195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.541079    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjr7\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-kube-api-access-jjjr7\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.541195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.541114    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.541797 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.541770    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.543398 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.543376    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.543499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.543424    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.543846 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.543824    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a76223c-2b65-4f5c-b7fc-cac2596c8899-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.543911 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.543832    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.552903 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.552876    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.553123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.553103    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjr7\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-kube-api-access-jjjr7\") pod \"istiod-openshift-gateway-7876bfcb74-r8jzx\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.582830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.582779    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"]
Oct 30 21:43:47.585950 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:43:47.585917    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915267bd_17be_4c23_a33b_9470d3b48394.slice/crio-285a0e3fc4a3761f3cdb83556b58681c3370bb2368ff8efae5b440381fe74cd0 WatchSource:0}: Error finding container 285a0e3fc4a3761f3cdb83556b58681c3370bb2368ff8efae5b440381fe74cd0: Status 404 returned error can't find the container with id 285a0e3fc4a3761f3cdb83556b58681c3370bb2368ff8efae5b440381fe74cd0
Oct 30 21:43:47.670784 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.670757    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:47.953164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:47.952959    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"]
Oct 30 21:43:48.456650 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:48.456608    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx" event={"ID":"7a76223c-2b65-4f5c-b7fc-cac2596c8899","Type":"ContainerStarted","Data":"bc0a5e1727ef295323c772c6799603cee25259563ed157861d8f32ed91f696f7"}
Oct 30 21:43:48.458037 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:48.458000    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs" event={"ID":"915267bd-17be-4c23-a33b-9470d3b48394","Type":"ContainerStarted","Data":"285a0e3fc4a3761f3cdb83556b58681c3370bb2368ff8efae5b440381fe74cd0"}
Oct 30 21:43:52.406440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:52.406403    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:43:52.406897 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:52.406492    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:43:53.489151 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:53.489114    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx" event={"ID":"7a76223c-2b65-4f5c-b7fc-cac2596c8899","Type":"ContainerStarted","Data":"3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d"}
Oct 30 21:43:53.489624 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:53.489246    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:53.490976 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:53.490947    4282 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-r8jzx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:43:53.491084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:53.490998    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx" podUID="7a76223c-2b65-4f5c-b7fc-cac2596c8899" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:43:53.518007 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:53.517439    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx" podStartSLOduration=2.071268221 podStartE2EDuration="6.517425177s" totalImagesPullingTime="4.446156956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:53.516594931 +0000 UTC m=+252.174691154" watchObservedRunningTime="2026-10-30 21:43:53.517425177 +0000 UTC m=+252.175521400"
Oct 30 21:43:54.493872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:54.493846    4282 generic.go:356] "Generic (PLEG): container finished" podID="915267bd-17be-4c23-a33b-9470d3b48394" containerID="4927359371ac7b4fb988bb67c943902ffdefe32c2ad0888144135505421ee64e" exitCode=0
Oct 30 21:43:54.494268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:54.493943    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs" event={"ID":"915267bd-17be-4c23-a33b-9470d3b48394","Type":"ContainerDied","Data":"4927359371ac7b4fb988bb67c943902ffdefe32c2ad0888144135505421ee64e"}
Oct 30 21:43:54.495611 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:54.495542    4282 generic.go:356] "Generic (PLEG): container finished" podID="e243a405-5977-4403-ad44-cbfa678a8743" containerID="a65348bf41b6b452a4369ca3d8dcc94db158c99ed11558459042f250452d2932" exitCode=0
Oct 30 21:43:54.495682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:54.495633    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl" event={"ID":"e243a405-5977-4403-ad44-cbfa678a8743","Type":"ContainerDied","Data":"a65348bf41b6b452a4369ca3d8dcc94db158c99ed11558459042f250452d2932"}
Oct 30 21:43:54.496445 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:54.496319    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:43:55.501252 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:55.501222    4282 generic.go:356] "Generic (PLEG): container finished" podID="915267bd-17be-4c23-a33b-9470d3b48394" containerID="89ca386127765df3956c913b11b9c1bab251fbcc4eeae5610a73acf25493b916" exitCode=0
Oct 30 21:43:55.501827 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:55.501338    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs" event={"ID":"915267bd-17be-4c23-a33b-9470d3b48394","Type":"ContainerDied","Data":"89ca386127765df3956c913b11b9c1bab251fbcc4eeae5610a73acf25493b916"}
Oct 30 21:43:57.512621 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:57.512577    4282 generic.go:356] "Generic (PLEG): container finished" podID="e243a405-5977-4403-ad44-cbfa678a8743" containerID="6f76804ff02cbfce5ce4c1432fc3a699a12aeb0ec5d14ce45a6e565d23cead4b" exitCode=0
Oct 30 21:43:57.513094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:43:57.512664    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl" event={"ID":"e243a405-5977-4403-ad44-cbfa678a8743","Type":"ContainerDied","Data":"6f76804ff02cbfce5ce4c1432fc3a699a12aeb0ec5d14ce45a6e565d23cead4b"}
Oct 30 21:44:02.537702 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:02.537662    4282 generic.go:356] "Generic (PLEG): container finished" podID="915267bd-17be-4c23-a33b-9470d3b48394" containerID="6e21e0275a140e06ad9225af70983f633bf18d3a5088cbeef5cc7d00392a1798" exitCode=0
Oct 30 21:44:02.538169 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:02.537710    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs" event={"ID":"915267bd-17be-4c23-a33b-9470d3b48394","Type":"ContainerDied","Data":"6e21e0275a140e06ad9225af70983f633bf18d3a5088cbeef5cc7d00392a1798"}
Oct 30 21:44:03.696777 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.696742    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:44:03.795988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.795914    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-util\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-util\") pod \"915267bd-17be-4c23-a33b-9470d3b48394\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") "
Oct 30 21:44:03.795988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.795986    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915267bd-17be-4c23-a33b-9470d3b48394-kube-api-access-4kg9t\" (UniqueName: \"kubernetes.io/projected/915267bd-17be-4c23-a33b-9470d3b48394-kube-api-access-4kg9t\") pod \"915267bd-17be-4c23-a33b-9470d3b48394\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") "
Oct 30 21:44:03.796220 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.796015    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-bundle\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-bundle\") pod \"915267bd-17be-4c23-a33b-9470d3b48394\" (UID: \"915267bd-17be-4c23-a33b-9470d3b48394\") "
Oct 30 21:44:03.796631 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.796603    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-bundle" pod "915267bd-17be-4c23-a33b-9470d3b48394" (UID: "915267bd-17be-4c23-a33b-9470d3b48394"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:03.798351 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.798330    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915267bd-17be-4c23-a33b-9470d3b48394-kube-api-access-4kg9t" pod "915267bd-17be-4c23-a33b-9470d3b48394" (UID: "915267bd-17be-4c23-a33b-9470d3b48394"). InnerVolumeSpecName "kube-api-access-4kg9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:03.799846 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.799817    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-util" pod "915267bd-17be-4c23-a33b-9470d3b48394" (UID: "915267bd-17be-4c23-a33b-9470d3b48394"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:03.896867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.896819    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kg9t\" (UniqueName: \"kubernetes.io/projected/915267bd-17be-4c23-a33b-9470d3b48394-kube-api-access-4kg9t\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:03.896867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.896846    4282 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-bundle\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:03.896867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:03.896860    4282 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/915267bd-17be-4c23-a33b-9470d3b48394-util\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:04.547009 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:04.546973    4282 generic.go:356] "Generic (PLEG): container finished" podID="e243a405-5977-4403-ad44-cbfa678a8743" containerID="2d2b9f2468b53275150d7b0411dcd3e487434c654132565faabb375db5b56ad4" exitCode=0
Oct 30 21:44:04.547188 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:04.547058    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl" event={"ID":"e243a405-5977-4403-ad44-cbfa678a8743","Type":"ContainerDied","Data":"2d2b9f2468b53275150d7b0411dcd3e487434c654132565faabb375db5b56ad4"}
Oct 30 21:44:04.548825 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:04.548802    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs" event={"ID":"915267bd-17be-4c23-a33b-9470d3b48394","Type":"ContainerDied","Data":"285a0e3fc4a3761f3cdb83556b58681c3370bb2368ff8efae5b440381fe74cd0"}
Oct 30 21:44:04.548947 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:04.548842    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71bd9zs"
Oct 30 21:44:04.548947 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:04.548843    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="285a0e3fc4a3761f3cdb83556b58681c3370bb2368ff8efae5b440381fe74cd0"
Oct 30 21:44:04.648757 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:44:04.648719    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:44:05.671022 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.670998    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:44:05.810328 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.810262    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-util\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-util\") pod \"e243a405-5977-4403-ad44-cbfa678a8743\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") "
Oct 30 21:44:05.810495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.810332    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e243a405-5977-4403-ad44-cbfa678a8743-kube-api-access-49f7w\" (UniqueName: \"kubernetes.io/projected/e243a405-5977-4403-ad44-cbfa678a8743-kube-api-access-49f7w\") pod \"e243a405-5977-4403-ad44-cbfa678a8743\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") "
Oct 30 21:44:05.810495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.810352    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-bundle\") pod \"e243a405-5977-4403-ad44-cbfa678a8743\" (UID: \"e243a405-5977-4403-ad44-cbfa678a8743\") "
Oct 30 21:44:05.810794 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.810772    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-bundle" pod "e243a405-5977-4403-ad44-cbfa678a8743" (UID: "e243a405-5977-4403-ad44-cbfa678a8743"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:05.812507 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.812463    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e243a405-5977-4403-ad44-cbfa678a8743-kube-api-access-49f7w" pod "e243a405-5977-4403-ad44-cbfa678a8743" (UID: "e243a405-5977-4403-ad44-cbfa678a8743"). InnerVolumeSpecName "kube-api-access-49f7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:44:05.814425 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.814402    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-util" pod "e243a405-5977-4403-ad44-cbfa678a8743" (UID: "e243a405-5977-4403-ad44-cbfa678a8743"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:44:05.911086 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.911047    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49f7w\" (UniqueName: \"kubernetes.io/projected/e243a405-5977-4403-ad44-cbfa678a8743-kube-api-access-49f7w\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:05.911086 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.911083    4282 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-bundle\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:05.911211 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:05.911097    4282 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e243a405-5977-4403-ad44-cbfa678a8743-util\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:44:06.557347 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:06.557308    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl" event={"ID":"e243a405-5977-4403-ad44-cbfa678a8743","Type":"ContainerDied","Data":"4bdb4d681170bb744a697fb2bf430b62c4542650cdca1ea9b6e59c077812ff6b"}
Oct 30 21:44:06.557347 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:06.557342    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447gpsl"
Oct 30 21:44:06.557577 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:06.557363    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bdb4d681170bb744a697fb2bf430b62c4542650cdca1ea9b6e59c077812ff6b"
Oct 30 21:44:25.296870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.296838    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"]
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297107    4282 cpu_manager.go:414] "containerMap: removing container" podUID="915267bd-17be-4c23-a33b-9470d3b48394" containerName="extract"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297118    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915267bd-17be-4c23-a33b-9470d3b48394" containerName="extract"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297137    4282 cpu_manager.go:414] "containerMap: removing container" podUID="e243a405-5977-4403-ad44-cbfa678a8743" containerName="pull"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297142    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e243a405-5977-4403-ad44-cbfa678a8743" containerName="pull"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297149    4282 cpu_manager.go:414] "containerMap: removing container" podUID="915267bd-17be-4c23-a33b-9470d3b48394" containerName="pull"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297154    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915267bd-17be-4c23-a33b-9470d3b48394" containerName="pull"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297167    4282 cpu_manager.go:414] "containerMap: removing container" podUID="e243a405-5977-4403-ad44-cbfa678a8743" containerName="util"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297171    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e243a405-5977-4403-ad44-cbfa678a8743" containerName="util"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297178    4282 cpu_manager.go:414] "containerMap: removing container" podUID="915267bd-17be-4c23-a33b-9470d3b48394" containerName="util"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297181    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915267bd-17be-4c23-a33b-9470d3b48394" containerName="util"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297186    4282 cpu_manager.go:414] "containerMap: removing container" podUID="e243a405-5977-4403-ad44-cbfa678a8743" containerName="extract"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297190    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e243a405-5977-4403-ad44-cbfa678a8743" containerName="extract"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297234    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e243a405-5977-4403-ad44-cbfa678a8743" containerName="extract"
Oct 30 21:44:25.297309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.297242    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="915267bd-17be-4c23-a33b-9470d3b48394" containerName="extract"
Oct 30 21:44:25.300163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.300147    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.330420 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.330387    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"]
Oct 30 21:44:25.378165 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.378135    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:44:25.378165 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.378148    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wwlz7\""
Oct 30 21:44:25.378165 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.378153    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:44:25.378347 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.378210    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:44:25.380702 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.380686    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:44:25.380776 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.380739    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:44:25.455871 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.455840    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.456009 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.455924    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.456009 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.455965    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fcr4\" (UniqueName: \"kubernetes.io/projected/c996fb42-08a2-4329-9d7d-9dee997ae0d2-kube-api-access-2fcr4\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.456080 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.456025    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c996fb42-08a2-4329-9d7d-9dee997ae0d2-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.456080 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.456069    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.456147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.456102    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c996fb42-08a2-4329-9d7d-9dee997ae0d2-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.556559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.556437    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c996fb42-08a2-4329-9d7d-9dee997ae0d2-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.556559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.556535    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.556559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.556561    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.556830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.556595    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fcr4\" (UniqueName: \"kubernetes.io/projected/c996fb42-08a2-4329-9d7d-9dee997ae0d2-kube-api-access-2fcr4\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.556830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.556625    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c996fb42-08a2-4329-9d7d-9dee997ae0d2-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.556830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.556672    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.557007 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.556982    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c996fb42-08a2-4329-9d7d-9dee997ae0d2-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.559126 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.559101    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c996fb42-08a2-4329-9d7d-9dee997ae0d2-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.559277 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.559262    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.559378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.559361    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.559510 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.559487    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c996fb42-08a2-4329-9d7d-9dee997ae0d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.569626 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.569600    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fcr4\" (UniqueName: \"kubernetes.io/projected/c996fb42-08a2-4329-9d7d-9dee997ae0d2-kube-api-access-2fcr4\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-qhqnw\" (UID: \"c996fb42-08a2-4329-9d7d-9dee997ae0d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.615553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.615530    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:25.924843 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:25.924812    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"]
Oct 30 21:44:25.927983 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:44:25.927956    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc996fb42_08a2_4329_9d7d_9dee997ae0d2.slice/crio-5f383fc9a38fdc097265349d1707ad8fcb9b352cbf7f7fcddf506cb9039e6874 WatchSource:0}: Error finding container 5f383fc9a38fdc097265349d1707ad8fcb9b352cbf7f7fcddf506cb9039e6874: Status 404 returned error can't find the container with id 5f383fc9a38fdc097265349d1707ad8fcb9b352cbf7f7fcddf506cb9039e6874
Oct 30 21:44:26.627668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:26.627632    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw" event={"ID":"c996fb42-08a2-4329-9d7d-9dee997ae0d2","Type":"ContainerStarted","Data":"5f383fc9a38fdc097265349d1707ad8fcb9b352cbf7f7fcddf506cb9039e6874"}
Oct 30 21:44:29.641908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:29.641825    4282 generic.go:356] "Generic (PLEG): container finished" podID="c996fb42-08a2-4329-9d7d-9dee997ae0d2" containerID="659b89b59b2b527f52df681982382363e4a184ad8fc5911d64859f13bbbf8211" exitCode=0
Oct 30 21:44:29.642325 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:29.641910    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw" event={"ID":"c996fb42-08a2-4329-9d7d-9dee997ae0d2","Type":"ContainerDied","Data":"659b89b59b2b527f52df681982382363e4a184ad8fc5911d64859f13bbbf8211"}
Oct 30 21:44:30.648444 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:30.648395    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw" event={"ID":"c996fb42-08a2-4329-9d7d-9dee997ae0d2","Type":"ContainerStarted","Data":"0e8ea182e57a06e003a6071e6e01fe91b39185c8521597e4db585583b681ecb8"}
Oct 30 21:44:30.648969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:30.648541    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:30.709104 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:30.709053    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw" podStartSLOduration=2.043007785 podStartE2EDuration="5.709040616s" totalImagesPullingTime="3.247244013s" totalInitContainerRuntime="418.788818ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:30.706244928 +0000 UTC m=+289.364341195" watchObservedRunningTime="2026-10-30 21:44:30.709040616 +0000 UTC m=+289.367136839"
Oct 30 21:44:34.647780 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:44:34.647740    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:44:41.653826 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:41.653797    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-qhqnw"
Oct 30 21:44:58.734279 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.734187    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"]
Oct 30 21:44:58.781322 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.781277    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"]
Oct 30 21:44:58.781565 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.781374    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.808042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.808018    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:44:58.808377 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.808360    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sz7sq\""
Oct 30 21:44:58.808430 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.808364    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:44:58.810348 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.810328    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59de6acd-8145-41f4-aa19-7f5696f7d790-tls-certs\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.810419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.810383    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59de6acd-8145-41f4-aa19-7f5696f7d790-tmp\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.810516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.810496    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4pv\" (UniqueName: \"kubernetes.io/projected/59de6acd-8145-41f4-aa19-7f5696f7d790-kube-api-access-rz4pv\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.911488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.911447    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59de6acd-8145-41f4-aa19-7f5696f7d790-tls-certs\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.911647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.911522    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59de6acd-8145-41f4-aa19-7f5696f7d790-tmp\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.911647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.911564    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4pv\" (UniqueName: \"kubernetes.io/projected/59de6acd-8145-41f4-aa19-7f5696f7d790-kube-api-access-rz4pv\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.913922 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.913895    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59de6acd-8145-41f4-aa19-7f5696f7d790-tmp\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.914034 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.914011    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59de6acd-8145-41f4-aa19-7f5696f7d790-tls-certs\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:58.934674 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:58.934652    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4pv\" (UniqueName: \"kubernetes.io/projected/59de6acd-8145-41f4-aa19-7f5696f7d790-kube-api-access-rz4pv\") pod \"kube-auth-proxy-78559bccc4-wjlbh\" (UID: \"59de6acd-8145-41f4-aa19-7f5696f7d790\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:59.110849 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:59.110758    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"
Oct 30 21:44:59.417166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:59.417142    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh"]
Oct 30 21:44:59.420168 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:44:59.420135    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59de6acd_8145_41f4_aa19_7f5696f7d790.slice/crio-17d422676b08d8e9bec393ded01ea5e43a29a1755bdb7e401475dcb9b9c8859f WatchSource:0}: Error finding container 17d422676b08d8e9bec393ded01ea5e43a29a1755bdb7e401475dcb9b9c8859f: Status 404 returned error can't find the container with id 17d422676b08d8e9bec393ded01ea5e43a29a1755bdb7e401475dcb9b9c8859f
Oct 30 21:44:59.422001 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:59.421982    4282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:44:59.761864 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:44:59.761776    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh" event={"ID":"59de6acd-8145-41f4-aa19-7f5696f7d790","Type":"ContainerStarted","Data":"17d422676b08d8e9bec393ded01ea5e43a29a1755bdb7e401475dcb9b9c8859f"}
Oct 30 21:45:02.577205 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.577170    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"]
Oct 30 21:45:02.580423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.580407    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.586212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.586180    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:45:02.586212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.586180    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:45:02.586382 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.586239    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-f7w45\""
Oct 30 21:45:02.593649 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.589941    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 21:45:02.593649 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.593395    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"]
Oct 30 21:45:02.642124 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.642095    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82a523cc-b6a3-4643-a0bf-aa2547f56eea-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.642124 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.642124    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82a523cc-b6a3-4643-a0bf-aa2547f56eea-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.642320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.642141    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82a523cc-b6a3-4643-a0bf-aa2547f56eea-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.642320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.642162    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5fw\" (UniqueName: \"kubernetes.io/projected/82a523cc-b6a3-4643-a0bf-aa2547f56eea-kube-api-access-nf5fw\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.743314 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.743282    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82a523cc-b6a3-4643-a0bf-aa2547f56eea-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.743498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.743322    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82a523cc-b6a3-4643-a0bf-aa2547f56eea-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.743498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.743347    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82a523cc-b6a3-4643-a0bf-aa2547f56eea-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.743498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.743379    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5fw\" (UniqueName: \"kubernetes.io/projected/82a523cc-b6a3-4643-a0bf-aa2547f56eea-kube-api-access-nf5fw\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.743772 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.743747    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82a523cc-b6a3-4643-a0bf-aa2547f56eea-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.743824 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.743756    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82a523cc-b6a3-4643-a0bf-aa2547f56eea-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.744211 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.744193    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82a523cc-b6a3-4643-a0bf-aa2547f56eea-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.757791 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.757770    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5fw\" (UniqueName: \"kubernetes.io/projected/82a523cc-b6a3-4643-a0bf-aa2547f56eea-kube-api-access-nf5fw\") pod \"kserve-module-controller-manager-7566cf5fc8-pnfdf\" (UID: \"82a523cc-b6a3-4643-a0bf-aa2547f56eea\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:02.778412 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.778385    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh" event={"ID":"59de6acd-8145-41f4-aa19-7f5696f7d790","Type":"ContainerStarted","Data":"1480490909db52b68b45de474c14a2c71d3cd5ec8badc70aa3b68e1309213a11"}
Oct 30 21:45:02.800040 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.799987    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78559bccc4-wjlbh" podStartSLOduration=2.332977251 podStartE2EDuration="4.799971439s" totalImagesPullingTime="2.466994188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.797608961 +0000 UTC m=+321.455705206" watchObservedRunningTime="2026-10-30 21:45:02.799971439 +0000 UTC m=+321.458067664"
Oct 30 21:45:02.896099 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:02.896021    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:03.176934 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:03.176907    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"]
Oct 30 21:45:03.180273 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:45:03.180247    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a523cc_b6a3_4643_a0bf_aa2547f56eea.slice/crio-44b0240bbf11d2634030e0bb7da827f6a447be36fd0f2f4e4841db81fba4c68c WatchSource:0}: Error finding container 44b0240bbf11d2634030e0bb7da827f6a447be36fd0f2f4e4841db81fba4c68c: Status 404 returned error can't find the container with id 44b0240bbf11d2634030e0bb7da827f6a447be36fd0f2f4e4841db81fba4c68c
Oct 30 21:45:03.785949 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:03.785908    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf" event={"ID":"82a523cc-b6a3-4643-a0bf-aa2547f56eea","Type":"ContainerStarted","Data":"44b0240bbf11d2634030e0bb7da827f6a447be36fd0f2f4e4841db81fba4c68c"}
Oct 30 21:45:04.654936 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:45:04.654903    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:45:04.791809 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:04.791778    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf" event={"ID":"82a523cc-b6a3-4643-a0bf-aa2547f56eea","Type":"ContainerStarted","Data":"b9338984764ae6fb7267c26068ecc84f0c7dbf3b40c65431996680096140df28"}
Oct 30 21:45:04.792194 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:04.791857    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:04.814814 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:04.814769    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf" podStartSLOduration=1.3591024649999999 podStartE2EDuration="2.814757034s" totalImagesPullingTime="1.455654569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.812953495 +0000 UTC m=+323.471049718" watchObservedRunningTime="2026-10-30 21:45:04.814757034 +0000 UTC m=+323.472853257"
Oct 30 21:45:25.797792 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:25.797762    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pnfdf"
Oct 30 21:45:34.647893 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:45:34.647851    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:45:39.842075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.842046    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:45:39.849194 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.849172    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:39.861165 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.861129    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:45:39.861294 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.861197    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 21:45:39.861294 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.861248    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:45:39.861649 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.861629    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-np89f\""
Oct 30 21:45:39.863323 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.863301    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:45:39.892384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.892355    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:45:39.943059 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.943034    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftwvh\" (UniqueName: \"kubernetes.io/projected/c648ae22-6494-4e9a-a4c9-bfc6e595c947-kube-api-access-ftwvh\") pod \"limitador-limitador-7c9b87d574-r9qsh\" (UID: \"c648ae22-6494-4e9a-a4c9-bfc6e595c947\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:39.943185 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:39.943071    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c648ae22-6494-4e9a-a4c9-bfc6e595c947-config-file\") pod \"limitador-limitador-7c9b87d574-r9qsh\" (UID: \"c648ae22-6494-4e9a-a4c9-bfc6e595c947\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:40.044243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:40.044205    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftwvh\" (UniqueName: \"kubernetes.io/projected/c648ae22-6494-4e9a-a4c9-bfc6e595c947-kube-api-access-ftwvh\") pod \"limitador-limitador-7c9b87d574-r9qsh\" (UID: \"c648ae22-6494-4e9a-a4c9-bfc6e595c947\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:40.044401 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:40.044255    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c648ae22-6494-4e9a-a4c9-bfc6e595c947-config-file\") pod \"limitador-limitador-7c9b87d574-r9qsh\" (UID: \"c648ae22-6494-4e9a-a4c9-bfc6e595c947\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:40.044959 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:40.044936    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c648ae22-6494-4e9a-a4c9-bfc6e595c947-config-file\") pod \"limitador-limitador-7c9b87d574-r9qsh\" (UID: \"c648ae22-6494-4e9a-a4c9-bfc6e595c947\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:40.056401 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:40.056374    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftwvh\" (UniqueName: \"kubernetes.io/projected/c648ae22-6494-4e9a-a4c9-bfc6e595c947-kube-api-access-ftwvh\") pod \"limitador-limitador-7c9b87d574-r9qsh\" (UID: \"c648ae22-6494-4e9a-a4c9-bfc6e595c947\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:40.162135 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:40.162048    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:40.437377 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:40.437351    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:45:40.440826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:45:40.440800    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc648ae22_6494_4e9a_a4c9_bfc6e595c947.slice/crio-e3e917a5b0f92d52992dab9d7ae6f96e508b8a1426674d14f753d9774ad30bb3 WatchSource:0}: Error finding container e3e917a5b0f92d52992dab9d7ae6f96e508b8a1426674d14f753d9774ad30bb3: Status 404 returned error can't find the container with id e3e917a5b0f92d52992dab9d7ae6f96e508b8a1426674d14f753d9774ad30bb3
Oct 30 21:45:40.933776 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:40.933749    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh" event={"ID":"c648ae22-6494-4e9a-a4c9-bfc6e595c947","Type":"ContainerStarted","Data":"e3e917a5b0f92d52992dab9d7ae6f96e508b8a1426674d14f753d9774ad30bb3"}
Oct 30 21:45:43.947577 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:43.947537    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh" event={"ID":"c648ae22-6494-4e9a-a4c9-bfc6e595c947","Type":"ContainerStarted","Data":"e2479bab5f48c68ef624342591e553b0d8fad17f4f734c3b7d4325fc575e9234"}
Oct 30 21:45:43.948061 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:43.947705    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:43.966256 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:43.966194    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh" podStartSLOduration=2.430773561 podStartE2EDuration="4.966175411s" totalImagesPullingTime="2.53540185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:43.964328741 +0000 UTC m=+362.622424965" watchObservedRunningTime="2026-10-30 21:45:43.966175411 +0000 UTC m=+362.624271635"
Oct 30 21:45:54.952326 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:54.952291    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"
Oct 30 21:45:56.980744 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.980709    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"]
Oct 30 21:45:56.983966 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.983948    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:56.986437 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.986418    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:56.986539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.986456    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e8949f-3858-4a3a-8028-00fc7a736f9f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:56.986583 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.986553    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:56.986629 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.986614    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:56.986716 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.986640    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxlz\" (UniqueName: \"kubernetes.io/projected/e4e8949f-3858-4a3a-8028-00fc7a736f9f-kube-api-access-kdxlz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:56.986716 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.986684    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:56.986792 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:56.986770    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.008094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.003347    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"]
Oct 30 21:45:57.087389 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.087354    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.087389 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.087391    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.087651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.087500    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e8949f-3858-4a3a-8028-00fc7a736f9f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.087651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.087572    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.087651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.087610    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.087889 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.087665    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxlz\" (UniqueName: \"kubernetes.io/projected/e4e8949f-3858-4a3a-8028-00fc7a736f9f-kube-api-access-kdxlz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.087889 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.087715    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.088346 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.088322    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.090166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.090146    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.090495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.090448    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.090579 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.090527    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e8949f-3858-4a3a-8028-00fc7a736f9f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.090695 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.090678    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e4e8949f-3858-4a3a-8028-00fc7a736f9f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.100491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.100444    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxlz\" (UniqueName: \"kubernetes.io/projected/e4e8949f-3858-4a3a-8028-00fc7a736f9f-kube-api-access-kdxlz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.101505 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.101457    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e8949f-3858-4a3a-8028-00fc7a736f9f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2clv5\" (UID: \"e4e8949f-3858-4a3a-8028-00fc7a736f9f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.295315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.295221    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:57.625098 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.625068    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"]
Oct 30 21:45:57.627128 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:45:57.627104    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e8949f_3858_4a3a_8028_00fc7a736f9f.slice/crio-c943822a7d3e182b28a213fc95dddf0f4f101770305d267eaefd376638de6b29 WatchSource:0}: Error finding container c943822a7d3e182b28a213fc95dddf0f4f101770305d267eaefd376638de6b29: Status 404 returned error can't find the container with id c943822a7d3e182b28a213fc95dddf0f4f101770305d267eaefd376638de6b29
Oct 30 21:45:57.629208 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.629178    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:45:57.629279 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:57.629236    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:45:58.003562 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:58.003525    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5" event={"ID":"e4e8949f-3858-4a3a-8028-00fc7a736f9f","Type":"ContainerStarted","Data":"184f9d4b6b725a1dd5226ca4aeaebe294845b7ffbb8db1b88d05315605c308ae"}
Oct 30 21:45:58.004021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:58.003582    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5" event={"ID":"e4e8949f-3858-4a3a-8028-00fc7a736f9f","Type":"ContainerStarted","Data":"c943822a7d3e182b28a213fc95dddf0f4f101770305d267eaefd376638de6b29"}
Oct 30 21:45:58.004021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:58.003785    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:58.005532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:58.005506    4282 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2clv5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:45:58.005655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:58.005570    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5" podUID="e4e8949f-3858-4a3a-8028-00fc7a736f9f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:45:58.032847 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:58.032181    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5" podStartSLOduration=2.032164997 podStartE2EDuration="2.032164997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:58.030625437 +0000 UTC m=+376.688721668" watchObservedRunningTime="2026-10-30 21:45:58.032164997 +0000 UTC m=+376.690261222"
Oct 30 21:45:59.008975 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.008947    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2clv5"
Oct 30 21:45:59.063239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.063205    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"]
Oct 30 21:45:59.063429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.063392    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx" podUID="7a76223c-2b65-4f5c-b7fc-cac2596c8899" containerName="discovery" containerID="cri-o://3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d" gracePeriod=30
Oct 30 21:45:59.310561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.310536    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:45:59.407301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.407263    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-cacerts\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-cacerts\") pod \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") "
Oct 30 21:45:59.407506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.407313    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-ca-configmap\") pod \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") "
Oct 30 21:45:59.407506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.407367    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a76223c-2b65-4f5c-b7fc-cac2596c8899-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a76223c-2b65-4f5c-b7fc-cac2596c8899-local-certs\") pod \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") "
Oct 30 21:45:59.407506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.407451    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-dns-cert\") pod \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") "
Oct 30 21:45:59.407506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.407497    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-token\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-token\") pod \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") "
Oct 30 21:45:59.407724 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.407537    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-kubeconfig\") pod \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") "
Oct 30 21:45:59.407724 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.407561    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-kube-api-access-jjjr7\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-kube-api-access-jjjr7\") pod \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\" (UID: \"7a76223c-2b65-4f5c-b7fc-cac2596c8899\") "
Oct 30 21:45:59.407860 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.407829    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-ca-configmap" pod "7a76223c-2b65-4f5c-b7fc-cac2596c8899" (UID: "7a76223c-2b65-4f5c-b7fc-cac2596c8899"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:59.410057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.410027    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-token" pod "7a76223c-2b65-4f5c-b7fc-cac2596c8899" (UID: "7a76223c-2b65-4f5c-b7fc-cac2596c8899"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:59.410057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.410046    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-kube-api-access-jjjr7" pod "7a76223c-2b65-4f5c-b7fc-cac2596c8899" (UID: "7a76223c-2b65-4f5c-b7fc-cac2596c8899"). InnerVolumeSpecName "kube-api-access-jjjr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:59.410228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.410150    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a76223c-2b65-4f5c-b7fc-cac2596c8899-local-certs" pod "7a76223c-2b65-4f5c-b7fc-cac2596c8899" (UID: "7a76223c-2b65-4f5c-b7fc-cac2596c8899"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:45:59.410228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.410179    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-dns-cert" pod "7a76223c-2b65-4f5c-b7fc-cac2596c8899" (UID: "7a76223c-2b65-4f5c-b7fc-cac2596c8899"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:59.410340 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.410224    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-cacerts" pod "7a76223c-2b65-4f5c-b7fc-cac2596c8899" (UID: "7a76223c-2b65-4f5c-b7fc-cac2596c8899"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:59.410340 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.410280    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-kubeconfig" pod "7a76223c-2b65-4f5c-b7fc-cac2596c8899" (UID: "7a76223c-2b65-4f5c-b7fc-cac2596c8899"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:45:59.509326 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.509292    4282 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-cacerts\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:59.509326 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.509327    4282 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-ca-configmap\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:59.509561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.509345    4282 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a76223c-2b65-4f5c-b7fc-cac2596c8899-local-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:59.509561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.509363    4282 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-csr-dns-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:59.509561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.509378    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:59.509561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.509392    4282 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a76223c-2b65-4f5c-b7fc-cac2596c8899-istio-kubeconfig\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:59.509561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:45:59.509406    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjr7\" (UniqueName: \"kubernetes.io/projected/7a76223c-2b65-4f5c-b7fc-cac2596c8899-kube-api-access-jjjr7\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:00.014454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.014420    4282 generic.go:356] "Generic (PLEG): container finished" podID="7a76223c-2b65-4f5c-b7fc-cac2596c8899" containerID="3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d" exitCode=0
Oct 30 21:46:00.014927 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.014494    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"
Oct 30 21:46:00.014927 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.014518    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx" event={"ID":"7a76223c-2b65-4f5c-b7fc-cac2596c8899","Type":"ContainerDied","Data":"3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d"}
Oct 30 21:46:00.014927 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.014572    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx" event={"ID":"7a76223c-2b65-4f5c-b7fc-cac2596c8899","Type":"ContainerDied","Data":"bc0a5e1727ef295323c772c6799603cee25259563ed157861d8f32ed91f696f7"}
Oct 30 21:46:00.014927 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.014594    4282 scope.go:122] "RemoveContainer" containerID="3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d"
Oct 30 21:46:00.023162 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.023143    4282 scope.go:122] "RemoveContainer" containerID="3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d"
Oct 30 21:46:00.023455 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:46:00.023431    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d\": container with ID starting with 3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d not found: ID does not exist" containerID="3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d"
Oct 30 21:46:00.023599 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.023478    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d"} err="failed to get container status \"3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d\": rpc error: code = NotFound desc = could not find container \"3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d\": container with ID starting with 3bdb07c815d447e86bb24a9f39968b2189e5061e92fb4babde24b2d9b1dd319d not found: ID does not exist"
Oct 30 21:46:00.031263 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.031228    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"]
Oct 30 21:46:00.035177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:00.035153    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-r8jzx"]
Oct 30 21:46:01.467157 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:01.467127    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a76223c-2b65-4f5c-b7fc-cac2596c8899" path="/var/lib/kubelet/pods/7a76223c-2b65-4f5c-b7fc-cac2596c8899/volumes"
Oct 30 21:46:04.647532 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:46:04.647497    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:46:24.598841 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.598757    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"]
Oct 30 21:46:24.600361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.599129    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7a76223c-2b65-4f5c-b7fc-cac2596c8899" containerName="discovery"
Oct 30 21:46:24.600361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.599145    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a76223c-2b65-4f5c-b7fc-cac2596c8899" containerName="discovery"
Oct 30 21:46:24.600361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.599213    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a76223c-2b65-4f5c-b7fc-cac2596c8899" containerName="discovery"
Oct 30 21:46:24.602252 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.602236    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.607807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.607770    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 21:46:24.607942 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.607923    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 21:46:24.609240 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.609206    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 21:46:24.609240 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.609240    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-4sxgc\""
Oct 30 21:46:24.633288 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.633265    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"]
Oct 30 21:46:24.699565 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.699537    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.699709 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.699594    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.699752 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.699709    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjs9\" (UniqueName: \"kubernetes.io/projected/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-kube-api-access-qjjs9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.699795 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.699749    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.800224 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.800188    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjs9\" (UniqueName: \"kubernetes.io/projected/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-kube-api-access-qjjs9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.800387 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.800236    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.800387 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.800274    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.800387 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.800316    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.800937 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.800912    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.801208 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.801186    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.802814 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.802797    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.811768 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.811744    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjs9\" (UniqueName: \"kubernetes.io/projected/655934c6-c911-4bbd-bc6e-7ce2dd3b368e-kube-api-access-qjjs9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-7hk96\" (UID: \"655934c6-c911-4bbd-bc6e-7ce2dd3b368e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:24.914420 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:24.914337    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:25.205178 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:25.205149    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"]
Oct 30 21:46:25.208459 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:46:25.208434    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655934c6_c911_4bbd_bc6e_7ce2dd3b368e.slice/crio-4b545bbde0643df62cb65bfb86c95cb505cde195287ed007dd649eb145b8ef49 WatchSource:0}: Error finding container 4b545bbde0643df62cb65bfb86c95cb505cde195287ed007dd649eb145b8ef49: Status 404 returned error can't find the container with id 4b545bbde0643df62cb65bfb86c95cb505cde195287ed007dd649eb145b8ef49
Oct 30 21:46:26.118403 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:26.118372    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96" event={"ID":"655934c6-c911-4bbd-bc6e-7ce2dd3b368e","Type":"ContainerStarted","Data":"4b545bbde0643df62cb65bfb86c95cb505cde195287ed007dd649eb145b8ef49"}
Oct 30 21:46:34.653069 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:46:34.653028    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:46:37.173383 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:37.173344    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96" event={"ID":"655934c6-c911-4bbd-bc6e-7ce2dd3b368e","Type":"ContainerStarted","Data":"46f4be3f64f0fe66ee86ed4ddc6166e40218b54ca1861478d8748d611ebb7574"}
Oct 30 21:46:37.173801 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:37.173429    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:37.193602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:37.193535    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96" podStartSLOduration=2.184034606 podStartE2EDuration="13.193517872s" totalImagesPullingTime="11.009483266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:37.192112134 +0000 UTC m=+415.850208356" watchObservedRunningTime="2026-10-30 21:46:37.193517872 +0000 UTC m=+415.851614094"
Oct 30 21:46:48.183984 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:48.183955    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-7hk96"
Oct 30 21:46:57.335956 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:46:57.335916    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:47:04.647702 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:47:04.647669    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:47:08.667817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.667786    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"]
Oct 30 21:47:08.714394 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.714363    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"]
Oct 30 21:47:08.714607 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.714458    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.723583 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.723562    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 21:47:08.723669 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.723624    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:47:08.723919 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.723905    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rm8tr\""
Oct 30 21:47:08.724312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.724300    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:47:08.725209 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.725197    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:47:08.765768 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.765741    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zjxm\" (UniqueName: \"kubernetes.io/projected/445638dc-b7cd-4036-a8f8-83ed400affa8-kube-api-access-8zjxm\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.765870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.765790    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445638dc-b7cd-4036-a8f8-83ed400affa8-maas-api-tls\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.765919 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.765870    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/445638dc-b7cd-4036-a8f8-83ed400affa8-openshift-service-ca\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.765919 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.765891    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/445638dc-b7cd-4036-a8f8-83ed400affa8-metrics-certs\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.866556 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.866522    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zjxm\" (UniqueName: \"kubernetes.io/projected/445638dc-b7cd-4036-a8f8-83ed400affa8-kube-api-access-8zjxm\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.866696 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.866573    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445638dc-b7cd-4036-a8f8-83ed400affa8-maas-api-tls\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.866696 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.866686    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/445638dc-b7cd-4036-a8f8-83ed400affa8-openshift-service-ca\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.866801 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.866724    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/445638dc-b7cd-4036-a8f8-83ed400affa8-metrics-certs\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.867624 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.867600    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/445638dc-b7cd-4036-a8f8-83ed400affa8-openshift-service-ca\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.869810 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.869792    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/445638dc-b7cd-4036-a8f8-83ed400affa8-maas-api-tls\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.869999 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.869979    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/445638dc-b7cd-4036-a8f8-83ed400affa8-metrics-certs\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:08.878442 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:08.878419    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zjxm\" (UniqueName: \"kubernetes.io/projected/445638dc-b7cd-4036-a8f8-83ed400affa8-kube-api-access-8zjxm\") pod \"maas-api-dffdd5df6-kgddg\" (UID: \"445638dc-b7cd-4036-a8f8-83ed400affa8\") " pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:09.026558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:09.026461    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:09.301648 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:09.301579    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"]
Oct 30 21:47:09.304826 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:47:09.304803    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445638dc_b7cd_4036_a8f8_83ed400affa8.slice/crio-4b4c89c4b156fc329c211ca82dae5ed1d5f0078ef3fa6c7fdb41083718dc96a0 WatchSource:0}: Error finding container 4b4c89c4b156fc329c211ca82dae5ed1d5f0078ef3fa6c7fdb41083718dc96a0: Status 404 returned error can't find the container with id 4b4c89c4b156fc329c211ca82dae5ed1d5f0078ef3fa6c7fdb41083718dc96a0
Oct 30 21:47:10.305626 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:10.305592    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg" event={"ID":"445638dc-b7cd-4036-a8f8-83ed400affa8","Type":"ContainerStarted","Data":"4b4c89c4b156fc329c211ca82dae5ed1d5f0078ef3fa6c7fdb41083718dc96a0"}
Oct 30 21:47:12.316394 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:12.316352    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg" event={"ID":"445638dc-b7cd-4036-a8f8-83ed400affa8","Type":"ContainerStarted","Data":"1cd7b921ce935d19738cace13a412793fd55f8ea0dc6ddefe1053f6d133f3e69"}
Oct 30 21:47:12.316839 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:12.316537    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:12.338833 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:12.338793    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg" podStartSLOduration=2.003023034 podStartE2EDuration="4.338781723s" totalImagesPullingTime="2.335758689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:12.33647724 +0000 UTC m=+450.994573454" watchObservedRunningTime="2026-10-30 21:47:12.338781723 +0000 UTC m=+450.996877946"
Oct 30 21:47:18.327711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:47:18.327683    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dffdd5df6-kgddg"
Oct 30 21:47:34.648114 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:47:34.648077    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:48:01.048430 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.048349    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-869599b8cc-mkw2r"]
Oct 30 21:48:01.051649 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.051632    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:01.056248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.056227    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 21:48:01.056404 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.056387    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 21:48:01.056497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.056411    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-s65f5\""
Oct 30 21:48:01.061633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.061615    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-869599b8cc-mkw2r"]
Oct 30 21:48:01.086026 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.085991    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a-metrics-certs\") pod \"ai-gateway-controller-869599b8cc-mkw2r\" (UID: \"4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a\") " pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:01.086184 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.086056    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6f42\" (UniqueName: \"kubernetes.io/projected/4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a-kube-api-access-c6f42\") pod \"ai-gateway-controller-869599b8cc-mkw2r\" (UID: \"4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a\") " pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:01.187287 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.187256    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6f42\" (UniqueName: \"kubernetes.io/projected/4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a-kube-api-access-c6f42\") pod \"ai-gateway-controller-869599b8cc-mkw2r\" (UID: \"4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a\") " pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:01.187455 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.187325    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a-metrics-certs\") pod \"ai-gateway-controller-869599b8cc-mkw2r\" (UID: \"4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a\") " pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:01.189757 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.189734    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a-metrics-certs\") pod \"ai-gateway-controller-869599b8cc-mkw2r\" (UID: \"4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a\") " pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:01.198779 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.198756    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6f42\" (UniqueName: \"kubernetes.io/projected/4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a-kube-api-access-c6f42\") pod \"ai-gateway-controller-869599b8cc-mkw2r\" (UID: \"4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a\") " pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:01.364912 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.364821    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:01.639990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:01.639922    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-869599b8cc-mkw2r"]
Oct 30 21:48:01.643111 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:48:01.643085    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba1f36f_e586_4ba1_a3f5_7dfad8fd363a.slice/crio-8f3c749e9f5292b094ce52642b78ac3d6411e8c38f6b7729381af30b5f9657fb WatchSource:0}: Error finding container 8f3c749e9f5292b094ce52642b78ac3d6411e8c38f6b7729381af30b5f9657fb: Status 404 returned error can't find the container with id 8f3c749e9f5292b094ce52642b78ac3d6411e8c38f6b7729381af30b5f9657fb
Oct 30 21:48:02.524565 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:02.524528    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r" event={"ID":"4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a","Type":"ContainerStarted","Data":"8f3c749e9f5292b094ce52642b78ac3d6411e8c38f6b7729381af30b5f9657fb"}
Oct 30 21:48:03.529486 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:03.529433    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r" event={"ID":"4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a","Type":"ContainerStarted","Data":"2c8de989a3f9767649927ccb3f0821c3b0aab9105065dc368fe622b146cda8b7"}
Oct 30 21:48:03.529903 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:03.529518    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:03.547867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:03.547824    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r" podStartSLOduration=1.273688342 podStartE2EDuration="2.547808736s" totalImagesPullingTime="1.274120394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:03.546303205 +0000 UTC m=+502.204399428" watchObservedRunningTime="2026-10-30 21:48:03.547808736 +0000 UTC m=+502.205904969"
Oct 30 21:48:04.647310 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:48:04.647274    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:48:14.536743 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:14.536716    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-869599b8cc-mkw2r"
Oct 30 21:48:21.718432 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.718400    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"]
Oct 30 21:48:21.721849 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.721833    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.730149 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.730128    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 21:48:21.730293 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.730252    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 21:48:21.730358 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.730297    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 21:48:21.730358 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.730310    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4lw49\""
Oct 30 21:48:21.735868 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.735848    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"]
Oct 30 21:48:21.746523 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.746504    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.746650 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.746529    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rqc\" (UniqueName: \"kubernetes.io/projected/2141267f-7dad-4ced-9a09-ffcf9d7524a6-kube-api-access-f4rqc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.746650 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.746577    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2141267f-7dad-4ced-9a09-ffcf9d7524a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.746650 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.746625    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.746822 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.746720    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.746822 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.746740    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.847916 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.847875    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.848077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.847932    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.848077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.847964    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.848077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.848009    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.848077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.848034    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rqc\" (UniqueName: \"kubernetes.io/projected/2141267f-7dad-4ced-9a09-ffcf9d7524a6-kube-api-access-f4rqc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.848270 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.848107    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2141267f-7dad-4ced-9a09-ffcf9d7524a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.848355 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.848334    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.848411 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.848370    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.848411 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.848400    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.850356 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.850336    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2141267f-7dad-4ced-9a09-ffcf9d7524a6-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.850801 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.850781    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2141267f-7dad-4ced-9a09-ffcf9d7524a6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:21.867857 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:21.867832    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rqc\" (UniqueName: \"kubernetes.io/projected/2141267f-7dad-4ced-9a09-ffcf9d7524a6-kube-api-access-f4rqc\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5\" (UID: \"2141267f-7dad-4ced-9a09-ffcf9d7524a6\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:22.034481 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:22.034387    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:22.161693 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:22.161664    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:48:22.317395 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:22.317310    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"]
Oct 30 21:48:22.322625 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:48:22.322601    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2141267f_7dad_4ced_9a09_ffcf9d7524a6.slice/crio-1dd28cac5e18f01e247f7e911fe750a537b5f1df5b15cc11e1a4fed65d895634 WatchSource:0}: Error finding container 1dd28cac5e18f01e247f7e911fe750a537b5f1df5b15cc11e1a4fed65d895634: Status 404 returned error can't find the container with id 1dd28cac5e18f01e247f7e911fe750a537b5f1df5b15cc11e1a4fed65d895634
Oct 30 21:48:22.610344 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:22.610259    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5" event={"ID":"2141267f-7dad-4ced-9a09-ffcf9d7524a6","Type":"ContainerStarted","Data":"1dd28cac5e18f01e247f7e911fe750a537b5f1df5b15cc11e1a4fed65d895634"}
Oct 30 21:48:23.616386 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:23.616357    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5" event={"ID":"2141267f-7dad-4ced-9a09-ffcf9d7524a6","Type":"ContainerStarted","Data":"1b99e63c76f4c959089054c6f720e5fe9ebf06efc7288141b926a01c0d6f7f27"}
Oct 30 21:48:23.637352 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:23.637295    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5" podStartSLOduration=1.434642213 podStartE2EDuration="2.637274014s" totalImagesPullingTime="1.202631801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:23.63684268 +0000 UTC m=+522.294938902" watchObservedRunningTime="2026-10-30 21:48:23.637274014 +0000 UTC m=+522.295370238"
Oct 30 21:48:24.035195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:24.035152    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:25.735354 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:25.735323    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:48:26.380084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.380052    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"]
Oct 30 21:48:26.383622 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.383601    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.387053 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.387028    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 21:48:26.394799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.394776    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"]
Oct 30 21:48:26.488548 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.488508    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.488548 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.488547    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.488823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.488670    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.488823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.488717    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.488923 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.488891    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.488981 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.488927    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mww8q\" (UniqueName: \"kubernetes.io/projected/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-kube-api-access-mww8q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.589856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.589822    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.589856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.589860    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.590083 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.589979    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.590083 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.590013    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.590083 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.590066    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.590305 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.590084    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mww8q\" (UniqueName: \"kubernetes.io/projected/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-kube-api-access-mww8q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.590356 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.590297    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.590356 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.590328    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.590421 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.590350    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.592173 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.592148    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.592645 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.592624    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.602224 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.602199    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mww8q\" (UniqueName: \"kubernetes.io/projected/4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8-kube-api-access-mww8q\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-qh7rx\" (UID: \"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.696986 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.696961    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:26.973028 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:26.972959    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"]
Oct 30 21:48:26.975771 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:48:26.975744    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad8c14f_8ea8_4d5d_911e_9cddd2ecf6a8.slice/crio-5ad74dc6978eaa794cc9b21b50b898f6a52786660a9a84ef604d4bfc8cd50390 WatchSource:0}: Error finding container 5ad74dc6978eaa794cc9b21b50b898f6a52786660a9a84ef604d4bfc8cd50390: Status 404 returned error can't find the container with id 5ad74dc6978eaa794cc9b21b50b898f6a52786660a9a84ef604d4bfc8cd50390
Oct 30 21:48:27.634187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:27.634110    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx" event={"ID":"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8","Type":"ContainerStarted","Data":"8e161dcf91aa5be851d7af771e2cc9f677d254ffc660b1d2f6450b7e8dc982fc"}
Oct 30 21:48:27.634187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:27.634159    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx" event={"ID":"4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8","Type":"ContainerStarted","Data":"5ad74dc6978eaa794cc9b21b50b898f6a52786660a9a84ef604d4bfc8cd50390"}
Oct 30 21:48:27.653036 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:27.652993    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx" podStartSLOduration=1.38071297 podStartE2EDuration="1.652979291s" totalImagesPullingTime="272.266321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:27.651593464 +0000 UTC m=+526.309689686" watchObservedRunningTime="2026-10-30 21:48:27.652979291 +0000 UTC m=+526.311075513"
Oct 30 21:48:27.697843 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:27.697816    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:28.105369 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.105334    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:48:28.431072 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.431045    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"]
Oct 30 21:48:28.434546 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.434528    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.442620 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.442596    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 21:48:28.447584 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.447560    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"]
Oct 30 21:48:28.504616 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.504583    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.504761 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.504685    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.504761 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.504733    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.504847 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.504772    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6p5c\" (UniqueName: \"kubernetes.io/projected/76d92e07-0511-4c67-abdc-bdc12605768b-kube-api-access-h6p5c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.504847 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.504828    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.504847 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.504844    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d92e07-0511-4c67-abdc-bdc12605768b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.605898 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.605851    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.606081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.605924    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.606081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.605968    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6p5c\" (UniqueName: \"kubernetes.io/projected/76d92e07-0511-4c67-abdc-bdc12605768b-kube-api-access-h6p5c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.606081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.605999    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.606081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.606020    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d92e07-0511-4c67-abdc-bdc12605768b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.606304 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.606180    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.606362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.606317    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.606362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.606326    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.606556 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.606532    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.608361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.608337    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d92e07-0511-4c67-abdc-bdc12605768b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.608634 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.608616    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d92e07-0511-4c67-abdc-bdc12605768b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.619214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.619193    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6p5c\" (UniqueName: \"kubernetes.io/projected/76d92e07-0511-4c67-abdc-bdc12605768b-kube-api-access-h6p5c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc\" (UID: \"76d92e07-0511-4c67-abdc-bdc12605768b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:28.747933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:28.747857    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:29.027386 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:29.027305    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"]
Oct 30 21:48:29.030459 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:48:29.030428    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d92e07_0511_4c67_abdc_bdc12605768b.slice/crio-66a76209e01575b156a93f9b9ff5f4de74159c22f52e0b225308fa497b6c3c38 WatchSource:0}: Error finding container 66a76209e01575b156a93f9b9ff5f4de74159c22f52e0b225308fa497b6c3c38: Status 404 returned error can't find the container with id 66a76209e01575b156a93f9b9ff5f4de74159c22f52e0b225308fa497b6c3c38
Oct 30 21:48:29.644259 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:29.644186    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc" event={"ID":"76d92e07-0511-4c67-abdc-bdc12605768b","Type":"ContainerStarted","Data":"7815540236ac74e7feb50bf5d8528830e04f8d4fb0cf497dc3c9b97e48167ca6"}
Oct 30 21:48:29.644259 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:29.644240    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc" event={"ID":"76d92e07-0511-4c67-abdc-bdc12605768b","Type":"ContainerStarted","Data":"66a76209e01575b156a93f9b9ff5f4de74159c22f52e0b225308fa497b6c3c38"}
Oct 30 21:48:29.667877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:29.667578    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc" podStartSLOduration=1.389365467 podStartE2EDuration="1.667560954s" totalImagesPullingTime="278.195487ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:29.665322926 +0000 UTC m=+528.323419149" watchObservedRunningTime="2026-10-30 21:48:29.667560954 +0000 UTC m=+528.325657179"
Oct 30 21:48:29.748117 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:29.748083    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:31.071285 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:31.071251    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:48:32.035398 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:32.035367    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:32.047749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:32.047728    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:32.669238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:32.669211    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5"
Oct 30 21:48:32.914153 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:32.914123    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"]
Oct 30 21:48:32.917766 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:32.917746    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:32.922387 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:32.922328    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 21:48:32.932767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:32.932742    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"]
Oct 30 21:48:33.045601 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.045567    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.045747 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.045621    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.045747 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.045699    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.045747 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.045742    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt8df\" (UniqueName: \"kubernetes.io/projected/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-kube-api-access-zt8df\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.045878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.045769    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.045878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.045810    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.146637 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.146599    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.146795 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.146658    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.146795 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.146713    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.146795 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.146739    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.146795 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.146770    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt8df\" (UniqueName: \"kubernetes.io/projected/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-kube-api-access-zt8df\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.146973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.146798    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.147067 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.147041    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.147133 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.147072    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.147229 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.147211    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.149122 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.149099    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.149271 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.149255    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.159849 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.159829    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt8df\" (UniqueName: \"kubernetes.io/projected/6e3b6173-4aa6-4e05-8829-0b6a47cccd38-kube-api-access-zt8df\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c\" (UID: \"6e3b6173-4aa6-4e05-8829-0b6a47cccd38\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.231017 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.230943    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:33.509897 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.509822    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"]
Oct 30 21:48:33.512940 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:48:33.512911    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3b6173_4aa6_4e05_8829_0b6a47cccd38.slice/crio-69ca938da81a9868ec38dd9b1b99ae2c4d0641bd1d899d29266f934c752f6f38 WatchSource:0}: Error finding container 69ca938da81a9868ec38dd9b1b99ae2c4d0641bd1d899d29266f934c752f6f38: Status 404 returned error can't find the container with id 69ca938da81a9868ec38dd9b1b99ae2c4d0641bd1d899d29266f934c752f6f38
Oct 30 21:48:33.663516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:33.663454    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c" event={"ID":"6e3b6173-4aa6-4e05-8829-0b6a47cccd38","Type":"ContainerStarted","Data":"69ca938da81a9868ec38dd9b1b99ae2c4d0641bd1d899d29266f934c752f6f38"}
Oct 30 21:48:34.610538 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:34.610501    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:48:34.648922 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:48:34.648882    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:48:34.669385 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:34.669354    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c" event={"ID":"6e3b6173-4aa6-4e05-8829-0b6a47cccd38","Type":"ContainerStarted","Data":"418d3a2946195fcf68afb1663b180f44f173e211c8b421702ca0df36caa1cd36"}
Oct 30 21:48:34.689863 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:34.689807    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c" podStartSLOduration=2.404260554 podStartE2EDuration="2.689745416s" totalImagesPullingTime="285.484862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:34.687565668 +0000 UTC m=+533.345661891" watchObservedRunningTime="2026-10-30 21:48:34.689745416 +0000 UTC m=+533.347841638"
Oct 30 21:48:35.231436 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:35.231378    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:36.698004 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:36.697974    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:36.710587 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:36.710565    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:37.692428 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:37.692398    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-qh7rx"
Oct 30 21:48:38.748962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:38.748931    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:38.761424 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:38.761401    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:39.068945 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:39.068866    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:48:39.701273 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:39.701249    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc"
Oct 30 21:48:43.231390 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:43.231358    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:43.243758 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:43.243739    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:43.719868 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:43.719843    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c"
Oct 30 21:48:53.957730 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:53.957696    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"]
Oct 30 21:48:53.964150 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:53.964132    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:53.978541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:53.978517    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"]
Oct 30 21:48:54.014721 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.014696    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kx6b6\""
Oct 30 21:48:54.122135 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122104    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.122312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122140    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.122312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122159    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/8c079c2b-d4bf-4a25-b5fb-437d434060fe-kube-api-access-z4mrs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.122312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122247    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.122312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122296    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.122549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122335    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.122549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122394    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.122549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122534    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.122688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.122582    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224099 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224031    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224099 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224085    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224286 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224129    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224286 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224166    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224286 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224190    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/8c079c2b-d4bf-4a25-b5fb-437d434060fe-kube-api-access-z4mrs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224286 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224248    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224595 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224273    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224595 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224520    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224595 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224525    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224595 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224576    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224744 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224606    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224854 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224827    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.224854 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.224845    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.225088 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.225069    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.226816 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.226795    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.226892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.226824    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.252964 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.252940    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c079c2b-d4bf-4a25-b5fb-437d434060fe-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.253675 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.253653    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mrs\" (UniqueName: \"kubernetes.io/projected/8c079c2b-d4bf-4a25-b5fb-437d434060fe-kube-api-access-z4mrs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-vgddj\" (UID: \"8c079c2b-d4bf-4a25-b5fb-437d434060fe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.277734 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.277712    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:54.563016 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.562997    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"]
Oct 30 21:48:54.564959 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:48:54.564934    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c079c2b_d4bf_4a25_b5fb_437d434060fe.slice/crio-53260044a7259a6ac0a5779854a228f01e4e959a3d9e1859aafeaa502ffe1b59 WatchSource:0}: Error finding container 53260044a7259a6ac0a5779854a228f01e4e959a3d9e1859aafeaa502ffe1b59: Status 404 returned error can't find the container with id 53260044a7259a6ac0a5779854a228f01e4e959a3d9e1859aafeaa502ffe1b59
Oct 30 21:48:54.756079 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:54.756043    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" event={"ID":"8c079c2b-d4bf-4a25-b5fb-437d434060fe","Type":"ContainerStarted","Data":"53260044a7259a6ac0a5779854a228f01e4e959a3d9e1859aafeaa502ffe1b59"}
Oct 30 21:48:57.611351 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:57.611317    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:48:57.611666 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:57.611392    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:48:57.611666 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:57.611423    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:48:57.772749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:57.772713    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" event={"ID":"8c079c2b-d4bf-4a25-b5fb-437d434060fe","Type":"ContainerStarted","Data":"2ebd57a7548ed2ce5d2111873826c6f90160fc84c404b12289aa58a590c9fd25"}
Oct 30 21:48:57.795990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:57.795935    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" podStartSLOduration=1.752220375 podStartE2EDuration="4.795920985s" totalImagesPullingTime="3.04370061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:57.794621645 +0000 UTC m=+556.452717867" watchObservedRunningTime="2026-10-30 21:48:57.795920985 +0000 UTC m=+556.454017208"
Oct 30 21:48:58.278237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:58.278195    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:48:58.280034 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:58.280007    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vgddj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:48:58.280198 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:58.280057    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" podUID="8c079c2b-d4bf-4a25-b5fb-437d434060fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:48:59.278813 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:59.278779    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vgddj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:48:59.279222 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:48:59.278833    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" podUID="8c079c2b-d4bf-4a25-b5fb-437d434060fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:49:00.278202 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:00.278151    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vgddj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:49:00.278419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:00.278214    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" podUID="8c079c2b-d4bf-4a25-b5fb-437d434060fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:49:01.278644 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:01.278606    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vgddj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:49:01.279132 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:01.278679    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" podUID="8c079c2b-d4bf-4a25-b5fb-437d434060fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:49:02.278359 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:02.278318    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vgddj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:49:02.280552 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:02.278387    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" podUID="8c079c2b-d4bf-4a25-b5fb-437d434060fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:49:03.278582 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:03.278539    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vgddj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:49:03.278759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:03.278622    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" podUID="8c079c2b-d4bf-4a25-b5fb-437d434060fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:49:04.278423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:04.278390    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-vgddj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 30 21:49:04.278838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:04.278486    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj" podUID="8c079c2b-d4bf-4a25-b5fb-437d434060fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 30 21:49:04.648916 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:49:04.648828    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:49:05.732767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:05.732734    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:49:05.733233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:05.733083    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:49:05.734002 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:05.733980    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-vgddj"
Oct 30 21:49:08.986879 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:08.986846    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"]
Oct 30 21:49:08.990724 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:08.990702    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.006835 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.006812    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"]
Oct 30 21:49:09.030867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.030842    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"]
Oct 30 21:49:09.034317 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.034300    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.048649 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.048626    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"]
Oct 30 21:49:09.059049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059030    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059152 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059059    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.059152 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059079    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgz47\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-kube-api-access-mgz47\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059152 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059101    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059202    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2fdf70-3c30-47a0-8f37-116848115739-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.059258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059226    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2fdf70-3c30-47a0-8f37-116848115739-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.059258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059248    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059347 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059275    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059347 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059329    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.059410 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059366    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059443 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059415    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059443 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059433    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.059519 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059496    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059552 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059518    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.059552 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059544    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7zkb\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-kube-api-access-v7zkb\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.059613 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059599    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.059645 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059629    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.059712 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.059698    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.160661 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.160634    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2fdf70-3c30-47a0-8f37-116848115739-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.160773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.160666    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.160773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.160685    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.160773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.160714    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.160773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.160747    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.160954 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.160779    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161006 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.160954    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161006 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.160997    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161109 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161034    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161109 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161080    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7zkb\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-kube-api-access-v7zkb\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161209 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161128    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161209 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161128    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161209 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161154    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161209 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161188    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161209 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161189    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161257    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161296    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161327    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgz47\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-kube-api-access-mgz47\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161360    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161388    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161394    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2fdf70-3c30-47a0-8f37-116848115739-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161504    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161521    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161540    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2fdf70-3c30-47a0-8f37-116848115739-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161610    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.161969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161857    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.161969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.161903    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.162286 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.162185    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.163393 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.163373    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.164288 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.164267    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2fdf70-3c30-47a0-8f37-116848115739-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.164479 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.164436    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.164651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.164636    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.172988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.172963    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.173167 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.173145    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7zkb\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-kube-api-access-v7zkb\") pod \"maas-default-gateway-openshift-default-85544d4cd7-dh2j9\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.173385 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.173364    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgz47\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-kube-api-access-mgz47\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.173540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.173522    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-mwpd9\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.303916 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.303858    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:09.346605 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.346572    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:09.584065 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.584040    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"]
Oct 30 21:49:09.586782 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:09.586755    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2fdf70_3c30_47a0_8f37_116848115739.slice/crio-2310563b8b16d30c30594cf151bd255d2be0ae76ffcb12e37ce1a01afb371400 WatchSource:0}: Error finding container 2310563b8b16d30c30594cf151bd255d2be0ae76ffcb12e37ce1a01afb371400: Status 404 returned error can't find the container with id 2310563b8b16d30c30594cf151bd255d2be0ae76ffcb12e37ce1a01afb371400
Oct 30 21:49:09.589778 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.589746    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:09.589853 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.589817    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:09.589853 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.589845    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:09.639895 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.639735    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"]
Oct 30 21:49:09.642108 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:09.642059    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e5d66c_b1e7_4e49_99c7_1b0fb33598fb.slice/crio-1d74ea6d9e7aaed0e4d5e248a11517d70a9cb4a1b1e459ff88b69dbf0fc84ef2 WatchSource:0}: Error finding container 1d74ea6d9e7aaed0e4d5e248a11517d70a9cb4a1b1e459ff88b69dbf0fc84ef2: Status 404 returned error can't find the container with id 1d74ea6d9e7aaed0e4d5e248a11517d70a9cb4a1b1e459ff88b69dbf0fc84ef2
Oct 30 21:49:09.645223 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.645193    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:09.645298 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.645257    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:09.645298 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.645292    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:09.840752 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.840613    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" event={"ID":"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb","Type":"ContainerStarted","Data":"aaf9086352a7ea6b8412c49cf3b0b08aafb05fc4a42b6636005de0e40b558c0a"}
Oct 30 21:49:09.841571 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.840980    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" event={"ID":"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb","Type":"ContainerStarted","Data":"1d74ea6d9e7aaed0e4d5e248a11517d70a9cb4a1b1e459ff88b69dbf0fc84ef2"}
Oct 30 21:49:09.842633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.842607    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" event={"ID":"6b2fdf70-3c30-47a0-8f37-116848115739","Type":"ContainerStarted","Data":"4037dfde7c46dd9f1b416eb730c4c5a03c250c4b0100a98fc365bacb2f8f944f"}
Oct 30 21:49:09.842749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.842658    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" event={"ID":"6b2fdf70-3c30-47a0-8f37-116848115739","Type":"ContainerStarted","Data":"2310563b8b16d30c30594cf151bd255d2be0ae76ffcb12e37ce1a01afb371400"}
Oct 30 21:49:09.864115 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.864066    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podStartSLOduration=1.8640493249999999 podStartE2EDuration="1.864049325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:09.861720257 +0000 UTC m=+568.519816480" watchObservedRunningTime="2026-10-30 21:49:09.864049325 +0000 UTC m=+568.522145600"
Oct 30 21:49:09.883187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:09.883138    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podStartSLOduration=1.883127024 podStartE2EDuration="1.883127024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:09.882108515 +0000 UTC m=+568.540204738" watchObservedRunningTime="2026-10-30 21:49:09.883127024 +0000 UTC m=+568.541223248"
Oct 30 21:49:10.304586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:10.304550    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:10.306315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:10.306290    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:49:10.306444 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:10.306357    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:49:10.347387 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:10.347361    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:10.348757 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:10.348732    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-mwpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:49:10.348858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:10.348786    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 21:49:11.305253 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:11.305205    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:49:11.305783 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:11.305262    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:49:11.347555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:11.347531    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-mwpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:49:11.347698 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:11.347573    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 21:49:12.305007 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:12.304965    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:49:12.305195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:12.305016    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:49:12.347750 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:12.347720    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-mwpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:49:12.348126 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:12.347768    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 21:49:13.304969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:13.304937    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:49:13.305147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:13.304993    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:49:13.347427 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:13.347395    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-mwpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:49:13.347613 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:13.347448    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 21:49:14.304983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:14.304941    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:49:14.305395 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:14.304999    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:49:14.347575 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:14.347540    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-mwpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:49:14.347726 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:14.347593    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 21:49:15.305073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:15.305034    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:49:15.305504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:15.305094    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:49:15.347662 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:15.347620    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-mwpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:49:15.347789 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:15.347673    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 21:49:16.304647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:16.304608    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 21:49:16.304856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:16.304655    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 21:49:16.347078 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:16.347049    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-mwpd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 21:49:16.347618 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:16.347094    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 21:49:18.009971 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:18.009928    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:18.010483 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:18.010425    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:18.011431 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:18.011411    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 21:49:18.016066 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:18.016044    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:18.016812 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:18.016794    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:18.017540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:18.017523    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 21:49:29.762905 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.762873    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"]
Oct 30 21:49:29.769405 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.769385    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.782320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.782296    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"]
Oct 30 21:49:29.789166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.789140    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-31a18d9b-gw-openshift-default-dockercfg-7dbk7\""
Oct 30 21:49:29.883883 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.883847    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-envoy\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.884042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.883900    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-token\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.884042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.883995    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-socket\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.884042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.884023    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-podinfo\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.884159 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.884060    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xszx4\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-kube-api-access-xszx4\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.884196 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.884158    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-credential-socket\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.884245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.884229    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istiod-ca-cert\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.884283 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.884259    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-data\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.884324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.884289    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-certs\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985146 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985110    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xszx4\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-kube-api-access-xszx4\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985146 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985147    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-credential-socket\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985183    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istiod-ca-cert\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985345    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-data\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985447 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985376    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-certs\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985447 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985419    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-envoy\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985449    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-token\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985510    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-socket\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985535    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-podinfo\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985663 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985644    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-credential-socket\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985777 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985752    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-data\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.985880 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.985782    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-certs\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.986036 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.986003    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istiod-ca-cert\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.986036 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.986026    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-socket\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.988004 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.987980    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-envoy\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.988117 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.988056    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-podinfo\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.998417 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.998390    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-token\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:29.998562 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:29.998545    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xszx4\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-kube-api-access-xszx4\") pod \"e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") " pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:30.084122 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:30.084037    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:30.375556 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:30.375531    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"]
Oct 30 21:49:30.378506 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:30.378460    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f33f6c_6ea7_4bde_8bb0_37c882ef7236.slice/crio-dd0bb305f77de47bbe89d4f0beebdbdab4f6198ae518f9dbefaba667ca687d7c WatchSource:0}: Error finding container dd0bb305f77de47bbe89d4f0beebdbdab4f6198ae518f9dbefaba667ca687d7c: Status 404 returned error can't find the container with id dd0bb305f77de47bbe89d4f0beebdbdab4f6198ae518f9dbefaba667ca687d7c
Oct 30 21:49:30.380606 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:30.380577    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:30.380671 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:30.380642    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:30.380710 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:30.380672    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:49:30.983941 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:30.983902    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc" event={"ID":"70f33f6c-6ea7-4bde-8bb0-37c882ef7236","Type":"ContainerStarted","Data":"15e132574c6468202d8700fe872ecaa4dcf7c27897df85ac4a0464d753ebd28f"}
Oct 30 21:49:30.984384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:30.983960    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc" event={"ID":"70f33f6c-6ea7-4bde-8bb0-37c882ef7236","Type":"ContainerStarted","Data":"dd0bb305f77de47bbe89d4f0beebdbdab4f6198ae518f9dbefaba667ca687d7c"}
Oct 30 21:49:31.006518 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:31.006446    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc" podStartSLOduration=2.00643149 podStartE2EDuration="2.00643149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:31.004818562 +0000 UTC m=+589.662914775" watchObservedRunningTime="2026-10-30 21:49:31.00643149 +0000 UTC m=+589.664527713"
Oct 30 21:49:31.085163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:31.085140    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:31.090061 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:31.090040    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:31.989432 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:31.989402    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:31.990407 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:31.990387    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:49:33.036159 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.036130    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"]
Oct 30 21:49:33.040231 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.040209    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.045038 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.045016    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-31a18d9b-serving-cert\""
Oct 30 21:49:33.050819 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.050790    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"]
Oct 30 21:49:33.116530 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.116445    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.116687 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.116535    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-metrics-certs\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.116739 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.116701    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-maas-api-tls\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.116739 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.116725    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5csgz\" (UniqueName: \"kubernetes.io/projected/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-kube-api-access-5csgz\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.217450 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.217417    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-maas-api-tls\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.217450 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.217450    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5csgz\" (UniqueName: \"kubernetes.io/projected/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-kube-api-access-5csgz\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.217713 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.217562    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.217713 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.217580    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-metrics-certs\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.218270 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.218251    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-openshift-service-ca\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.220027 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.220001    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-maas-api-tls\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.220123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.220081    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-metrics-certs\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.229879 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.229852    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5csgz\" (UniqueName: \"kubernetes.io/projected/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-kube-api-access-5csgz\") pod \"maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.362799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.362714    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:33.689587 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.689555    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"]
Oct 30 21:49:33.693315 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:33.693291    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf81a9aa_b7e9_4cf7_b085_a5d1210ad5a5.slice/crio-81bb5006d996820dea8406801447f38766b2ad89c16961705f12017827e4bc0d WatchSource:0}: Error finding container 81bb5006d996820dea8406801447f38766b2ad89c16961705f12017827e4bc0d: Status 404 returned error can't find the container with id 81bb5006d996820dea8406801447f38766b2ad89c16961705f12017827e4bc0d
Oct 30 21:49:33.999176 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:33.999092    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8" event={"ID":"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5","Type":"ContainerStarted","Data":"81bb5006d996820dea8406801447f38766b2ad89c16961705f12017827e4bc0d"}
Oct 30 21:49:34.648647 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:49:34.648567    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:49:34.697910 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.697883    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"]
Oct 30 21:49:34.701520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.701498    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:34.708177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.708147    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-681bcb24-dockercfg-cb6db\""
Oct 30 21:49:34.708584 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.708564    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-681bcb24\""
Oct 30 21:49:34.720496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.720437    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"]
Oct 30 21:49:34.722870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.722850    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"]
Oct 30 21:49:34.726506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.726482    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:34.742711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.742690    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"]
Oct 30 21:49:34.836912 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.836866    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tpk\" (UniqueName: \"kubernetes.io/projected/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-kube-api-access-l5tpk\") pod \"payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs\" (UID: \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:34.837137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.836975    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fd6t\" (UniqueName: \"kubernetes.io/projected/da9755c2-d890-46e6-a83b-db0bd1ee9387-kube-api-access-9fd6t\") pod \"payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j\" (UID: \"da9755c2-d890-46e6-a83b-db0bd1ee9387\") " pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:34.837137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.837002    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da9755c2-d890-46e6-a83b-db0bd1ee9387-plugins-config-volume\") pod \"payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j\" (UID: \"da9755c2-d890-46e6-a83b-db0bd1ee9387\") " pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:34.837137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.837046    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs\" (UID: \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:34.938070 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.938041    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fd6t\" (UniqueName: \"kubernetes.io/projected/da9755c2-d890-46e6-a83b-db0bd1ee9387-kube-api-access-9fd6t\") pod \"payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j\" (UID: \"da9755c2-d890-46e6-a83b-db0bd1ee9387\") " pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:34.938248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.938076    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da9755c2-d890-46e6-a83b-db0bd1ee9387-plugins-config-volume\") pod \"payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j\" (UID: \"da9755c2-d890-46e6-a83b-db0bd1ee9387\") " pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:34.938248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.938106    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs\" (UID: \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:34.938248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.938164    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tpk\" (UniqueName: \"kubernetes.io/projected/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-kube-api-access-l5tpk\") pod \"payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs\" (UID: \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:34.938808 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.938791    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da9755c2-d890-46e6-a83b-db0bd1ee9387-plugins-config-volume\") pod \"payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j\" (UID: \"da9755c2-d890-46e6-a83b-db0bd1ee9387\") " pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:34.938918 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.938900    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs\" (UID: \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:34.960858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.960828    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tpk\" (UniqueName: \"kubernetes.io/projected/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-kube-api-access-l5tpk\") pod \"payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs\" (UID: \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:34.962936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:34.962918    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fd6t\" (UniqueName: \"kubernetes.io/projected/da9755c2-d890-46e6-a83b-db0bd1ee9387-kube-api-access-9fd6t\") pod \"payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j\" (UID: \"da9755c2-d890-46e6-a83b-db0bd1ee9387\") " pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:35.005534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.005506    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8" event={"ID":"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5","Type":"ContainerStarted","Data":"2a9d67f99ef3cabb81dc5ff0c82ad358e38cee000af9546d7497d6ff9a3d2a8c"}
Oct 30 21:49:35.005674 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.005568    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:35.014693 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.014670    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:35.039826 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.039767    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8" podStartSLOduration=1.519192088 podStartE2EDuration="2.039667771s" totalImagesPullingTime="520.475683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:35.035835305 +0000 UTC m=+593.693931531" watchObservedRunningTime="2026-10-30 21:49:35.039667771 +0000 UTC m=+593.697763994"
Oct 30 21:49:35.041555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.041528    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:35.296449 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.296343    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"]
Oct 30 21:49:35.300113 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:35.300087    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fb3c6f_e8fb_4cf6_8eaa_8241388d8161.slice/crio-8409b834c169e30dced9219615655db17f5c95ceb1574a1987fda9603e57d6cb WatchSource:0}: Error finding container 8409b834c169e30dced9219615655db17f5c95ceb1574a1987fda9603e57d6cb: Status 404 returned error can't find the container with id 8409b834c169e30dced9219615655db17f5c95ceb1574a1987fda9603e57d6cb
Oct 30 21:49:35.339846 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.339814    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"]
Oct 30 21:49:35.341037 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:35.341012    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9755c2_d890_46e6_a83b_db0bd1ee9387.slice/crio-a5c0e700988e33ee6264c876bbf66aedd4fe4c47192e9d05e99dfb11aae96370 WatchSource:0}: Error finding container a5c0e700988e33ee6264c876bbf66aedd4fe4c47192e9d05e99dfb11aae96370: Status 404 returned error can't find the container with id a5c0e700988e33ee6264c876bbf66aedd4fe4c47192e9d05e99dfb11aae96370
Oct 30 21:49:35.415634 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.415603    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:49:35.712044 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.712002    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"]
Oct 30 21:49:35.724437 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.724352    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.743628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.743600    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9528ebfc-serving-cert\""
Oct 30 21:49:35.746911 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.746885    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"]
Oct 30 21:49:35.852786 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.852748    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzjq\" (UniqueName: \"kubernetes.io/projected/89ace07b-1c82-4f9f-a223-ffc2576e452d-kube-api-access-5jzjq\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.852786 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.852800    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-maas-api-tls\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.853036 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.852876    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-metrics-certs\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.853036 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.852994    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ace07b-1c82-4f9f-a223-ffc2576e452d-openshift-service-ca\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.954828 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.954788    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzjq\" (UniqueName: \"kubernetes.io/projected/89ace07b-1c82-4f9f-a223-ffc2576e452d-kube-api-access-5jzjq\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.955027 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.954859    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-maas-api-tls\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.955027 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.954902    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-metrics-certs\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.955027 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.954938    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ace07b-1c82-4f9f-a223-ffc2576e452d-openshift-service-ca\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.955652 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.955625    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ace07b-1c82-4f9f-a223-ffc2576e452d-openshift-service-ca\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.979089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.963358    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-metrics-certs\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.979089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.968179    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-maas-api-tls\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:35.979089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:35.971433    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzjq\" (UniqueName: \"kubernetes.io/projected/89ace07b-1c82-4f9f-a223-ffc2576e452d-kube-api-access-5jzjq\") pod \"maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:36.036969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:36.036415    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j" event={"ID":"da9755c2-d890-46e6-a83b-db0bd1ee9387","Type":"ContainerStarted","Data":"a5c0e700988e33ee6264c876bbf66aedd4fe4c47192e9d05e99dfb11aae96370"}
Oct 30 21:49:36.042946 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:36.042844    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs" event={"ID":"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161","Type":"ContainerStarted","Data":"8409b834c169e30dced9219615655db17f5c95ceb1574a1987fda9603e57d6cb"}
Oct 30 21:49:36.053785 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:36.053489    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:36.370364 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:36.370281    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"]
Oct 30 21:49:36.383457 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:36.383419    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ace07b_1c82_4f9f_a223_ffc2576e452d.slice/crio-a90014db9749e7023ec4f872fcce47db5c1dd5faf1145a5483a2aa9f05477423 WatchSource:0}: Error finding container a90014db9749e7023ec4f872fcce47db5c1dd5faf1145a5483a2aa9f05477423: Status 404 returned error can't find the container with id a90014db9749e7023ec4f872fcce47db5c1dd5faf1145a5483a2aa9f05477423
Oct 30 21:49:37.050360 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:37.050322    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf" event={"ID":"89ace07b-1c82-4f9f-a223-ffc2576e452d","Type":"ContainerStarted","Data":"da9f3da0ba89783067d8b3044c56caee345c5c0315923e538c7f923fe97b09d1"}
Oct 30 21:49:37.050870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:37.050387    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf" event={"ID":"89ace07b-1c82-4f9f-a223-ffc2576e452d","Type":"ContainerStarted","Data":"a90014db9749e7023ec4f872fcce47db5c1dd5faf1145a5483a2aa9f05477423"}
Oct 30 21:49:37.050870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:37.050517    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:37.082506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:37.082363    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf" podStartSLOduration=1.6443851010000001 podStartE2EDuration="2.082344416s" totalImagesPullingTime="437.959315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:37.075193633 +0000 UTC m=+595.733289857" watchObservedRunningTime="2026-10-30 21:49:37.082344416 +0000 UTC m=+595.740440640"
Oct 30 21:49:38.385453 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.385373    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"]
Oct 30 21:49:38.389050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.389029    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.397715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.397697    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-681bcb24-serving-cert\""
Oct 30 21:49:38.464521 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.464495    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"]
Oct 30 21:49:38.490895 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.490868    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-maas-api-tls\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.490994 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.490897    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4540d21-3990-4340-b98d-13b13c25d174-openshift-service-ca\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.490994 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.490919    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489xx\" (UniqueName: \"kubernetes.io/projected/b4540d21-3990-4340-b98d-13b13c25d174-kube-api-access-489xx\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.491133 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.491113    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-metrics-certs\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.592637 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.592600    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-metrics-certs\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.592807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.592646    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-maas-api-tls\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.592807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.592674    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4540d21-3990-4340-b98d-13b13c25d174-openshift-service-ca\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.592807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.592704    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489xx\" (UniqueName: \"kubernetes.io/projected/b4540d21-3990-4340-b98d-13b13c25d174-kube-api-access-489xx\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.593339 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.593317    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4540d21-3990-4340-b98d-13b13c25d174-openshift-service-ca\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.595164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.595143    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-maas-api-tls\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.595227 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.595181    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-metrics-certs\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.616513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.616485    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-489xx\" (UniqueName: \"kubernetes.io/projected/b4540d21-3990-4340-b98d-13b13c25d174-kube-api-access-489xx\") pod \"maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.702500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.702461    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:38.987224 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:38.987191    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"]
Oct 30 21:49:38.989623 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:38.989599    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4540d21_3990_4340_b98d_13b13c25d174.slice/crio-3ed446d272c73eb7c726f1e2ee32da7182b30ad9c62c2059b6eee5cd2b3584be WatchSource:0}: Error finding container 3ed446d272c73eb7c726f1e2ee32da7182b30ad9c62c2059b6eee5cd2b3584be: Status 404 returned error can't find the container with id 3ed446d272c73eb7c726f1e2ee32da7182b30ad9c62c2059b6eee5cd2b3584be
Oct 30 21:49:39.062074 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:39.062039    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd" event={"ID":"b4540d21-3990-4340-b98d-13b13c25d174","Type":"ContainerStarted","Data":"3ed446d272c73eb7c726f1e2ee32da7182b30ad9c62c2059b6eee5cd2b3584be"}
Oct 30 21:49:39.063660 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:39.063632    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j" event={"ID":"da9755c2-d890-46e6-a83b-db0bd1ee9387","Type":"ContainerStarted","Data":"4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2"}
Oct 30 21:49:39.063812 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:39.063757    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:39.065752 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:39.065723    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs" event={"ID":"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161","Type":"ContainerStarted","Data":"361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325"}
Oct 30 21:49:39.066877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:39.066854    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:39.098541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:39.098485    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j" podStartSLOduration=2.39600581 podStartE2EDuration="5.098449157s" totalImagesPullingTime="2.702443347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.095459724 +0000 UTC m=+597.753555958" watchObservedRunningTime="2026-10-30 21:49:39.098449157 +0000 UTC m=+597.756545384"
Oct 30 21:49:39.121667 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:39.121625    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs" podStartSLOduration=2.448701172 podStartE2EDuration="5.121612834s" totalImagesPullingTime="2.672911662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.119981293 +0000 UTC m=+597.778077526" watchObservedRunningTime="2026-10-30 21:49:39.121612834 +0000 UTC m=+597.779709086"
Oct 30 21:49:40.071950 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:40.071908    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd" event={"ID":"b4540d21-3990-4340-b98d-13b13c25d174","Type":"ContainerStarted","Data":"618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822"}
Oct 30 21:49:40.072353 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:40.072014    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:41.048116 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:41.048084    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:49:41.072491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:41.072412    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd" podStartSLOduration=2.622869356 podStartE2EDuration="3.072399571s" totalImagesPullingTime="449.530215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:40.096828473 +0000 UTC m=+598.754924707" watchObservedRunningTime="2026-10-30 21:49:41.072399571 +0000 UTC m=+599.730495819"
Oct 30 21:49:43.061238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:43.061210    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:49:44.113694 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:44.113666    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:49:45.817509 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.816538    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"]
Oct 30 21:49:45.832905 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.832877    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"]
Oct 30 21:49:45.833064 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.832995    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.840712 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.840692    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4487d16a-serving-cert\""
Oct 30 21:49:45.879933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.879908    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-openshift-service-ca\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.880068 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.879953    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.880068 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.879978    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d6c7\" (UniqueName: \"kubernetes.io/projected/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-kube-api-access-6d6c7\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.880149 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.880079    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-metrics-certs\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.981017 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.980968    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-openshift-service-ca\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.981216 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.981066    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.981216 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.981112    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d6c7\" (UniqueName: \"kubernetes.io/projected/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-kube-api-access-6d6c7\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.981216 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.981176    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-metrics-certs\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.981417 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:49:45.981215    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-serving-cert: secret "maas-api-e2e-derive-4487d16a-serving-cert" not found
Oct 30 21:49:45.981417 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:49:45.981290    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls podName:d0d4f9b0-d28f-4021-b9cd-e30732452bfc nodeName:}" failed. No retries permitted until 2026-10-30 21:49:46.481272413 +0000 UTC m=+605.139368635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls") pod "maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2" (UID: "d0d4f9b0-d28f-4021-b9cd-e30732452bfc") : secret "maas-api-e2e-derive-4487d16a-serving-cert" not found
Oct 30 21:49:45.981772 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.981749    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-openshift-service-ca\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.984032 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.984010    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-metrics-certs\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:45.993722 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:45.993703    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d6c7\" (UniqueName: \"kubernetes.io/projected/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-kube-api-access-6d6c7\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:46.083280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:46.083215    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:49:46.490752 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:46.490722    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:46.493318 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:46.493293    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls\") pod \"maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:46.746461 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:46.746375    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:47.027412 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:47.027343    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"]
Oct 30 21:49:47.030683 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:49:47.030650    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d4f9b0_d28f_4021_b9cd_e30732452bfc.slice/crio-16c9d407ae2383a6ca7ffa10cf933f01ea460eb529d14397ad0f179b60527b0d WatchSource:0}: Error finding container 16c9d407ae2383a6ca7ffa10cf933f01ea460eb529d14397ad0f179b60527b0d: Status 404 returned error can't find the container with id 16c9d407ae2383a6ca7ffa10cf933f01ea460eb529d14397ad0f179b60527b0d
Oct 30 21:49:47.108731 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:47.108699    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2" event={"ID":"d0d4f9b0-d28f-4021-b9cd-e30732452bfc","Type":"ContainerStarted","Data":"16c9d407ae2383a6ca7ffa10cf933f01ea460eb529d14397ad0f179b60527b0d"}
Oct 30 21:49:48.115061 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:48.115020    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2" event={"ID":"d0d4f9b0-d28f-4021-b9cd-e30732452bfc","Type":"ContainerStarted","Data":"4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93"}
Oct 30 21:49:48.115456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:48.115156    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:49:48.135228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:48.135184    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2" podStartSLOduration=2.678924387 podStartE2EDuration="3.135170769s" totalImagesPullingTime="456.246382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:48.13456025 +0000 UTC m=+606.792656472" watchObservedRunningTime="2026-10-30 21:49:48.135170769 +0000 UTC m=+606.793266992"
Oct 30 21:49:50.075069 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:50.075044    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:49:50.075434 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:50.075105    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:49:53.708143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:53.708104    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:49:54.125830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:49:54.125755    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:50:03.464493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:03.459129    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"]
Oct 30 21:50:03.486190 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:03.486155    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"]
Oct 30 21:50:03.981503 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:03.981296    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"]
Oct 30 21:50:03.982852 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:03.982823    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf" podUID="89ace07b-1c82-4f9f-a223-ffc2576e452d" containerName="maas-api" containerID="cri-o://da9f3da0ba89783067d8b3044c56caee345c5c0315923e538c7f923fe97b09d1" gracePeriod=30
Oct 30 21:50:04.228236 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.228132    4282 generic.go:356] "Generic (PLEG): container finished" podID="89ace07b-1c82-4f9f-a223-ffc2576e452d" containerID="da9f3da0ba89783067d8b3044c56caee345c5c0315923e538c7f923fe97b09d1" exitCode=0
Oct 30 21:50:04.228236 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.228179    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf" event={"ID":"89ace07b-1c82-4f9f-a223-ffc2576e452d","Type":"ContainerDied","Data":"da9f3da0ba89783067d8b3044c56caee345c5c0315923e538c7f923fe97b09d1"}
Oct 30 21:50:04.306518 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.306497    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:50:04.413791 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.413757    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ace07b-1c82-4f9f-a223-ffc2576e452d-kube-api-access-5jzjq\" (UniqueName: \"kubernetes.io/projected/89ace07b-1c82-4f9f-a223-ffc2576e452d-kube-api-access-5jzjq\") pod \"89ace07b-1c82-4f9f-a223-ffc2576e452d\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") "
Oct 30 21:50:04.413969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.413806    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-maas-api-tls\") pod \"89ace07b-1c82-4f9f-a223-ffc2576e452d\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") "
Oct 30 21:50:04.413969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.413851    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-metrics-certs\") pod \"89ace07b-1c82-4f9f-a223-ffc2576e452d\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") "
Oct 30 21:50:04.413969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.413903    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ace07b-1c82-4f9f-a223-ffc2576e452d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ace07b-1c82-4f9f-a223-ffc2576e452d-openshift-service-ca\") pod \"89ace07b-1c82-4f9f-a223-ffc2576e452d\" (UID: \"89ace07b-1c82-4f9f-a223-ffc2576e452d\") "
Oct 30 21:50:04.414224 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.414205    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ace07b-1c82-4f9f-a223-ffc2576e452d-openshift-service-ca" pod "89ace07b-1c82-4f9f-a223-ffc2576e452d" (UID: "89ace07b-1c82-4f9f-a223-ffc2576e452d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:04.414340 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.414328    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ace07b-1c82-4f9f-a223-ffc2576e452d-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:04.416185 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.416156    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-maas-api-tls" pod "89ace07b-1c82-4f9f-a223-ffc2576e452d" (UID: "89ace07b-1c82-4f9f-a223-ffc2576e452d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:04.416185 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.416161    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ace07b-1c82-4f9f-a223-ffc2576e452d-kube-api-access-5jzjq" pod "89ace07b-1c82-4f9f-a223-ffc2576e452d" (UID: "89ace07b-1c82-4f9f-a223-ffc2576e452d"). InnerVolumeSpecName "kube-api-access-5jzjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:04.416323 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.416207    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-metrics-certs" pod "89ace07b-1c82-4f9f-a223-ffc2576e452d" (UID: "89ace07b-1c82-4f9f-a223-ffc2576e452d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:04.515223 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.515186    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jzjq\" (UniqueName: \"kubernetes.io/projected/89ace07b-1c82-4f9f-a223-ffc2576e452d-kube-api-access-5jzjq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:04.515223 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.515218    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:04.515223 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:04.515228    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ace07b-1c82-4f9f-a223-ffc2576e452d-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:04.648824 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:50:04.648742    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:50:05.037117 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.037084    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"]
Oct 30 21:50:05.037399 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.037371    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8" podUID="cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" containerName="maas-api" containerID="cri-o://2a9d67f99ef3cabb81dc5ff0c82ad358e38cee000af9546d7497d6ff9a3d2a8c" gracePeriod=30
Oct 30 21:50:05.235611 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.235583    4282 generic.go:356] "Generic (PLEG): container finished" podID="cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" containerID="2a9d67f99ef3cabb81dc5ff0c82ad358e38cee000af9546d7497d6ff9a3d2a8c" exitCode=0
Oct 30 21:50:05.235818 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.235673    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8" event={"ID":"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5","Type":"ContainerDied","Data":"2a9d67f99ef3cabb81dc5ff0c82ad358e38cee000af9546d7497d6ff9a3d2a8c"}
Oct 30 21:50:05.237292 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.237265    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf" event={"ID":"89ace07b-1c82-4f9f-a223-ffc2576e452d","Type":"ContainerDied","Data":"a90014db9749e7023ec4f872fcce47db5c1dd5faf1145a5483a2aa9f05477423"}
Oct 30 21:50:05.237455 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.237322    4282 scope.go:122] "RemoveContainer" containerID="da9f3da0ba89783067d8b3044c56caee345c5c0315923e538c7f923fe97b09d1"
Oct 30 21:50:05.237547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.237498    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"
Oct 30 21:50:05.290044 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.289956    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"]
Oct 30 21:50:05.290422 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.290369    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd" podUID="b4540d21-3990-4340-b98d-13b13c25d174" containerName="maas-api" containerID="cri-o://618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822" gracePeriod=30
Oct 30 21:50:05.302322 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.302292    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"]
Oct 30 21:50:05.315663 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.315633    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9528ebfc-5878b5fc9b-6spdf"]
Oct 30 21:50:05.423554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.423529    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:50:05.499018 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.498990    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ace07b-1c82-4f9f-a223-ffc2576e452d" path="/var/lib/kubelet/pods/89ace07b-1c82-4f9f-a223-ffc2576e452d/volumes"
Oct 30 21:50:05.501769 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.501748    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"]
Oct 30 21:50:05.501979 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.501956    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc" podUID="70f33f6c-6ea7-4bde-8bb0-37c882ef7236" containerName="istio-proxy" containerID="cri-o://15e132574c6468202d8700fe872ecaa4dcf7c27897df85ac4a0464d753ebd28f" gracePeriod=30
Oct 30 21:50:05.525566 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.525524    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-maas-api-tls\") pod \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") "
Oct 30 21:50:05.526042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.525630    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-openshift-service-ca\") pod \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") "
Oct 30 21:50:05.526042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.525676    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-metrics-certs\") pod \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") "
Oct 30 21:50:05.526042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.525737    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-kube-api-access-5csgz\" (UniqueName: \"kubernetes.io/projected/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-kube-api-access-5csgz\") pod \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\" (UID: \"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5\") "
Oct 30 21:50:05.526670 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.526613    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-openshift-service-ca" pod "cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" (UID: "cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:05.529691 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.529668    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-kube-api-access-5csgz" pod "cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" (UID: "cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5"). InnerVolumeSpecName "kube-api-access-5csgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:05.529774 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.529684    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-maas-api-tls" pod "cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" (UID: "cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:05.529774 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.529737    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-metrics-certs" pod "cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" (UID: "cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:05.628540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.627733    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:05.628540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.627772    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:05.628540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.627786    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5csgz\" (UniqueName: \"kubernetes.io/projected/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-kube-api-access-5csgz\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:05.628540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.627800    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:05.723258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.723224    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:50:05.829906 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.829873    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-metrics-certs\") pod \"b4540d21-3990-4340-b98d-13b13c25d174\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") "
Oct 30 21:50:05.830097 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.829964    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-maas-api-tls\") pod \"b4540d21-3990-4340-b98d-13b13c25d174\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") "
Oct 30 21:50:05.830097 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.830049    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4540d21-3990-4340-b98d-13b13c25d174-kube-api-access-489xx\" (UniqueName: \"kubernetes.io/projected/b4540d21-3990-4340-b98d-13b13c25d174-kube-api-access-489xx\") pod \"b4540d21-3990-4340-b98d-13b13c25d174\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") "
Oct 30 21:50:05.830206 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.830140    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4540d21-3990-4340-b98d-13b13c25d174-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4540d21-3990-4340-b98d-13b13c25d174-openshift-service-ca\") pod \"b4540d21-3990-4340-b98d-13b13c25d174\" (UID: \"b4540d21-3990-4340-b98d-13b13c25d174\") "
Oct 30 21:50:05.830600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.830567    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4540d21-3990-4340-b98d-13b13c25d174-openshift-service-ca" pod "b4540d21-3990-4340-b98d-13b13c25d174" (UID: "b4540d21-3990-4340-b98d-13b13c25d174"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:05.832148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.832118    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-maas-api-tls" pod "b4540d21-3990-4340-b98d-13b13c25d174" (UID: "b4540d21-3990-4340-b98d-13b13c25d174"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:05.832148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.832141    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-metrics-certs" pod "b4540d21-3990-4340-b98d-13b13c25d174" (UID: "b4540d21-3990-4340-b98d-13b13c25d174"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:05.832280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.832167    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4540d21-3990-4340-b98d-13b13c25d174-kube-api-access-489xx" pod "b4540d21-3990-4340-b98d-13b13c25d174" (UID: "b4540d21-3990-4340-b98d-13b13c25d174"). InnerVolumeSpecName "kube-api-access-489xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:05.931705 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.931678    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:05.931705 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.931704    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-489xx\" (UniqueName: \"kubernetes.io/projected/b4540d21-3990-4340-b98d-13b13c25d174-kube-api-access-489xx\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:05.931929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.931713    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4540d21-3990-4340-b98d-13b13c25d174-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:05.931929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:05.931723    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4540d21-3990-4340-b98d-13b13c25d174-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:06.243125 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.243037    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8" event={"ID":"cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5","Type":"ContainerDied","Data":"81bb5006d996820dea8406801447f38766b2ad89c16961705f12017827e4bc0d"}
Oct 30 21:50:06.243324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.243127    4282 scope.go:122] "RemoveContainer" containerID="2a9d67f99ef3cabb81dc5ff0c82ad358e38cee000af9546d7497d6ff9a3d2a8c"
Oct 30 21:50:06.243324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.243054    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"
Oct 30 21:50:06.245409 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.245381    4282 generic.go:356] "Generic (PLEG): container finished" podID="b4540d21-3990-4340-b98d-13b13c25d174" containerID="618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822" exitCode=0
Oct 30 21:50:06.245557 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.245414    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"
Oct 30 21:50:06.245557 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.245459    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd" event={"ID":"b4540d21-3990-4340-b98d-13b13c25d174","Type":"ContainerDied","Data":"618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822"}
Oct 30 21:50:06.245557 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.245517    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd" event={"ID":"b4540d21-3990-4340-b98d-13b13c25d174","Type":"ContainerDied","Data":"3ed446d272c73eb7c726f1e2ee32da7182b30ad9c62c2059b6eee5cd2b3584be"}
Oct 30 21:50:06.277872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.277790    4282 scope.go:122] "RemoveContainer" containerID="618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822"
Oct 30 21:50:06.286951 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.286930    4282 scope.go:122] "RemoveContainer" containerID="618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822"
Oct 30 21:50:06.287240 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:50:06.287223    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822\": container with ID starting with 618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822 not found: ID does not exist" containerID="618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822"
Oct 30 21:50:06.287280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.287250    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822"} err="failed to get container status \"618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822\": rpc error: code = NotFound desc = could not find container \"618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822\": container with ID starting with 618650c08bf4899946e28e007148aa57debcc250db03454bff4bc79227ed3822 not found: ID does not exist"
Oct 30 21:50:06.294741 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.294720    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"]
Oct 30 21:50:06.311099 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.311078    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-681bcb24-788ffcbb66-9ddgd"]
Oct 30 21:50:06.328569 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.328542    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"]
Oct 30 21:50:06.340628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.340601    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-31a18d9b-d45c5648d-xc9g8"]
Oct 30 21:50:06.449140 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449108    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"]
Oct 30 21:50:06.449532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449519    4282 cpu_manager.go:414] "containerMap: removing container" podUID="89ace07b-1c82-4f9f-a223-ffc2576e452d" containerName="maas-api"
Oct 30 21:50:06.449532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449533    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ace07b-1c82-4f9f-a223-ffc2576e452d" containerName="maas-api"
Oct 30 21:50:06.449629 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449556    4282 cpu_manager.go:414] "containerMap: removing container" podUID="cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" containerName="maas-api"
Oct 30 21:50:06.449629 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449565    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" containerName="maas-api"
Oct 30 21:50:06.449629 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449586    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b4540d21-3990-4340-b98d-13b13c25d174" containerName="maas-api"
Oct 30 21:50:06.449629 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449591    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4540d21-3990-4340-b98d-13b13c25d174" containerName="maas-api"
Oct 30 21:50:06.449748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449650    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ace07b-1c82-4f9f-a223-ffc2576e452d" containerName="maas-api"
Oct 30 21:50:06.449748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449661    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" containerName="maas-api"
Oct 30 21:50:06.449748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.449667    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4540d21-3990-4340-b98d-13b13c25d174" containerName="maas-api"
Oct 30 21:50:06.460491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.458392    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.481530 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.469369    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1529dc-openshift-default-dockercfg-zmbwx\""
Oct 30 21:50:06.485773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.485742    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"]
Oct 30 21:50:06.640101 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640008    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.640101 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640078    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.640612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640178    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.640612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640233    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.640612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640337    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.640612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640371    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.640612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640398    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.640612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640444    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b416f91-5c95-43cd-bdc6-402bca298e02-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.640612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.640529    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbpz\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-kube-api-access-hnbpz\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.741995 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.741962    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742008    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742034    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742145    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742189    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742233    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b416f91-5c95-43cd-bdc6-402bca298e02-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742270    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbpz\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-kube-api-access-hnbpz\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742337    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742398    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742410    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742453    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742665 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742495    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.742830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.742808    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.743034 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.743016    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b416f91-5c95-43cd-bdc6-402bca298e02-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.744635 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.744615    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.744728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.744696    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.768066 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.768007    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.781597 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.781571    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbpz\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-kube-api-access-hnbpz\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.786932 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.786911    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:06.888856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.868062    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"]
Oct 30 21:50:06.890951 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.889944    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"]
Oct 30 21:50:06.890951 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.890061    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.003495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:06.994103    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ceb36a-openshift-default-dockercfg-vtb7n\""
Oct 30 21:50:07.045554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045519    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d8a943-a45d-434a-a400-eec464164ea2-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.045732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045570    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.045732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045668    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.045837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045733    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.045837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045753    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d8a943-a45d-434a-a400-eec464164ea2-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.045837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045785    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.045837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045820    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr2fv\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-kube-api-access-cr2fv\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.046020 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045841    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.046020 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.045872    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.084374 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.084086    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"]
Oct 30 21:50:07.088183 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:50:07.088143    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b416f91_5c95_43cd_bdc6_402bca298e02.slice/crio-566cba97de91bf9be307b63a951c5b1a1945485def1ffe60b3dec7e3728671cf WatchSource:0}: Error finding container 566cba97de91bf9be307b63a951c5b1a1945485def1ffe60b3dec7e3728671cf: Status 404 returned error can't find the container with id 566cba97de91bf9be307b63a951c5b1a1945485def1ffe60b3dec7e3728671cf
Oct 30 21:50:07.089743 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.089707    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:07.089843 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.089789    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:07.089843 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.089837    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:07.147302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.146506    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr2fv\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-kube-api-access-cr2fv\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.146560    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.146612    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.146669    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d8a943-a45d-434a-a400-eec464164ea2-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.146695    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.146745    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147653 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.147514    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147653 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.147574    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147653 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.147604    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d8a943-a45d-434a-a400-eec464164ea2-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.147690    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.147848 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.147829    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.148001 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.147975    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d8a943-a45d-434a-a400-eec464164ea2-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.148170 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.148147    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.148296 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.148278    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.171491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.150797    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d8a943-a45d-434a-a400-eec464164ea2-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.178803 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.178724    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.192627 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.191237    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr2fv\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-kube-api-access-cr2fv\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.193601 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.193576    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.205201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.205182    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:07.238223 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.238183    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:50:07.299987 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.299928    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" event={"ID":"4b416f91-5c95-43cd-bdc6-402bca298e02","Type":"ContainerStarted","Data":"34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46"}
Oct 30 21:50:07.300174 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.300002    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" event={"ID":"4b416f91-5c95-43cd-bdc6-402bca298e02","Type":"ContainerStarted","Data":"566cba97de91bf9be307b63a951c5b1a1945485def1ffe60b3dec7e3728671cf"}
Oct 30 21:50:07.346389 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.346343    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podStartSLOduration=1.346328896 podStartE2EDuration="1.346328896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:07.343784916 +0000 UTC m=+626.001881142" watchObservedRunningTime="2026-10-30 21:50:07.346328896 +0000 UTC m=+626.004425118"
Oct 30 21:50:07.504143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.504108    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4540d21-3990-4340-b98d-13b13c25d174" path="/var/lib/kubelet/pods/b4540d21-3990-4340-b98d-13b13c25d174/volumes"
Oct 30 21:50:07.505089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.505067    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5" path="/var/lib/kubelet/pods/cf81a9aa-b7e9-4cf7-b085-a5d1210ad5a5/volumes"
Oct 30 21:50:07.596075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.594238    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"]
Oct 30 21:50:07.597010 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:50:07.596975    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d8a943_a45d_434a_a400_eec464164ea2.slice/crio-d618a8cc9e071886fbafea9d635ffa4c15f71a37026beaf8282b1055f823e842 WatchSource:0}: Error finding container d618a8cc9e071886fbafea9d635ffa4c15f71a37026beaf8282b1055f823e842: Status 404 returned error can't find the container with id d618a8cc9e071886fbafea9d635ffa4c15f71a37026beaf8282b1055f823e842
Oct 30 21:50:07.599555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.599514    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:07.599656 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.599608    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:07.599721 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.599682    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:07.787269 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:07.787182    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:08.280306 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.262170    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"]
Oct 30 21:50:08.280306 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.278024    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2" podUID="d0d4f9b0-d28f-4021-b9cd-e30732452bfc" containerName="maas-api" containerID="cri-o://4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93" gracePeriod=30
Oct 30 21:50:08.316668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.316641    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" event={"ID":"73d8a943-a45d-434a-a400-eec464164ea2","Type":"ContainerStarted","Data":"eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00"}
Oct 30 21:50:08.316869 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.316854    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" event={"ID":"73d8a943-a45d-434a-a400-eec464164ea2","Type":"ContainerStarted","Data":"d618a8cc9e071886fbafea9d635ffa4c15f71a37026beaf8282b1055f823e842"}
Oct 30 21:50:08.377614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.377554    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podStartSLOduration=2.377539762 podStartE2EDuration="2.377539762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:08.377290877 +0000 UTC m=+627.035387113" watchObservedRunningTime="2026-10-30 21:50:08.377539762 +0000 UTC m=+627.035635985"
Oct 30 21:50:08.470532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.469542    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j" podUID="da9755c2-d890-46e6-a83b-db0bd1ee9387" containerName="payload-processing" containerID="cri-o://4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2" gracePeriod=25
Oct 30 21:50:08.486783 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.486591    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs" podUID="71fb3c6f-e8fb-4cf6-8eaa-8241388d8161" containerName="payload-pre-processing" containerID="cri-o://361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325" gracePeriod=25
Oct 30 21:50:08.641731 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.641424    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:50:08.708251 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.708223    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-kube-api-access-6d6c7\" (UniqueName: \"kubernetes.io/projected/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-kube-api-access-6d6c7\") pod \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") "
Oct 30 21:50:08.708461 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.708302    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls\") pod \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") "
Oct 30 21:50:08.708461 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.708322    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-metrics-certs\") pod \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") "
Oct 30 21:50:08.708461 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.708372    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-openshift-service-ca\") pod \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\" (UID: \"d0d4f9b0-d28f-4021-b9cd-e30732452bfc\") "
Oct 30 21:50:08.709707 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.709645    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-openshift-service-ca" pod "d0d4f9b0-d28f-4021-b9cd-e30732452bfc" (UID: "d0d4f9b0-d28f-4021-b9cd-e30732452bfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:08.712205 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.711646    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls" pod "d0d4f9b0-d28f-4021-b9cd-e30732452bfc" (UID: "d0d4f9b0-d28f-4021-b9cd-e30732452bfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:08.712205 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.711897    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-metrics-certs" pod "d0d4f9b0-d28f-4021-b9cd-e30732452bfc" (UID: "d0d4f9b0-d28f-4021-b9cd-e30732452bfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:08.713080 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.713058    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-kube-api-access-6d6c7" pod "d0d4f9b0-d28f-4021-b9cd-e30732452bfc" (UID: "d0d4f9b0-d28f-4021-b9cd-e30732452bfc"). InnerVolumeSpecName "kube-api-access-6d6c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:08.815746 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.815256    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d6c7\" (UniqueName: \"kubernetes.io/projected/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-kube-api-access-6d6c7\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:08.815746 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.815302    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:08.815746 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.815320    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:08.815746 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.815334    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4f9b0-d28f-4021-b9cd-e30732452bfc-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:08.852755 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.852664    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:08.858524 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.858417    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:50:08.902539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.902514    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:50:08.921343 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.921313    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"]
Oct 30 21:50:08.922027 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922003    4282 cpu_manager.go:414] "containerMap: removing container" podUID="d0d4f9b0-d28f-4021-b9cd-e30732452bfc" containerName="maas-api"
Oct 30 21:50:08.922106 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922032    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d4f9b0-d28f-4021-b9cd-e30732452bfc" containerName="maas-api"
Oct 30 21:50:08.922106 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922060    4282 cpu_manager.go:414] "containerMap: removing container" podUID="da9755c2-d890-46e6-a83b-db0bd1ee9387" containerName="payload-processing"
Oct 30 21:50:08.922106 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922069    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9755c2-d890-46e6-a83b-db0bd1ee9387" containerName="payload-processing"
Oct 30 21:50:08.922106 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922096    4282 cpu_manager.go:414] "containerMap: removing container" podUID="71fb3c6f-e8fb-4cf6-8eaa-8241388d8161" containerName="payload-pre-processing"
Oct 30 21:50:08.922244 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922106    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fb3c6f-e8fb-4cf6-8eaa-8241388d8161" containerName="payload-pre-processing"
Oct 30 21:50:08.922244 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922228    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fb3c6f-e8fb-4cf6-8eaa-8241388d8161" containerName="payload-pre-processing"
Oct 30 21:50:08.922301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922247    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d4f9b0-d28f-4021-b9cd-e30732452bfc" containerName="maas-api"
Oct 30 21:50:08.922301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.922260    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9755c2-d890-46e6-a83b-db0bd1ee9387" containerName="payload-processing"
Oct 30 21:50:08.925784 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.925759    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:08.941374 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:08.941341    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"]
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.032367    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-939a16-openshift-default-dockercfg-6sr4l\""
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.032381    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9755c2-d890-46e6-a83b-db0bd1ee9387-kube-api-access-9fd6t\" (UniqueName: \"kubernetes.io/projected/da9755c2-d890-46e6-a83b-db0bd1ee9387-kube-api-access-9fd6t\") pod \"da9755c2-d890-46e6-a83b-db0bd1ee9387\" (UID: \"da9755c2-d890-46e6-a83b-db0bd1ee9387\") "
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.032634    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-kube-api-access-l5tpk\" (UniqueName: \"kubernetes.io/projected/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-kube-api-access-l5tpk\") pod \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\" (UID: \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\") "
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.032681    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9755c2-d890-46e6-a83b-db0bd1ee9387-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da9755c2-d890-46e6-a83b-db0bd1ee9387-plugins-config-volume\") pod \"da9755c2-d890-46e6-a83b-db0bd1ee9387\" (UID: \"da9755c2-d890-46e6-a83b-db0bd1ee9387\") "
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.032734    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-plugins-config-volume\") pod \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\" (UID: \"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161\") "
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033165    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-plugins-config-volume" pod "71fb3c6f-e8fb-4cf6-8eaa-8241388d8161" (UID: "71fb3c6f-e8fb-4cf6-8eaa-8241388d8161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033588    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a025f32-89c7-4009-8b77-7ee839ed8c20-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033631    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033658    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033690    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033867    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033892    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033913    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033941    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.033964    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcks\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-kube-api-access-lhcks\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.034003    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:09.034877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.034394    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9755c2-d890-46e6-a83b-db0bd1ee9387-plugins-config-volume" pod "da9755c2-d890-46e6-a83b-db0bd1ee9387" (UID: "da9755c2-d890-46e6-a83b-db0bd1ee9387"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:09.042273 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.042246    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9755c2-d890-46e6-a83b-db0bd1ee9387-kube-api-access-9fd6t" pod "da9755c2-d890-46e6-a83b-db0bd1ee9387" (UID: "da9755c2-d890-46e6-a83b-db0bd1ee9387"). InnerVolumeSpecName "kube-api-access-9fd6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:09.062773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.055112    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-kube-api-access-l5tpk" pod "71fb3c6f-e8fb-4cf6-8eaa-8241388d8161" (UID: "71fb3c6f-e8fb-4cf6-8eaa-8241388d8161"). InnerVolumeSpecName "kube-api-access-l5tpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:09.134809 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.134727    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.134809 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.134778    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.134812    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.134855    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.134881    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcks\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-kube-api-access-lhcks\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.134927    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a025f32-89c7-4009-8b77-7ee839ed8c20-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.134962    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.134989    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135021    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135094    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5tpk\" (UniqueName: \"kubernetes.io/projected/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161-kube-api-access-l5tpk\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:09.135301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135110    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da9755c2-d890-46e6-a83b-db0bd1ee9387-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:09.135301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135127    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fd6t\" (UniqueName: \"kubernetes.io/projected/da9755c2-d890-46e6-a83b-db0bd1ee9387-kube-api-access-9fd6t\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:09.135599 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135335    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135599 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135429    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135710 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135610    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135762 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135715    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a025f32-89c7-4009-8b77-7ee839ed8c20-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.135858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.135839    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.137599 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.137574    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.137681 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.137640    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.156143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.156121    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcks\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-kube-api-access-lhcks\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.166844 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.166822    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.205838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.205816    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:09.211014 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.210993    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:09.251077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.251041    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:09.325179 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.325153    4282 generic.go:356] "Generic (PLEG): container finished" podID="71fb3c6f-e8fb-4cf6-8eaa-8241388d8161" containerID="361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325" exitCode=0
Oct 30 21:50:09.325308 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.325200    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"
Oct 30 21:50:09.325308 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.325247    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs" event={"ID":"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161","Type":"ContainerDied","Data":"361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325"}
Oct 30 21:50:09.325308 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.325294    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs" event={"ID":"71fb3c6f-e8fb-4cf6-8eaa-8241388d8161","Type":"ContainerDied","Data":"8409b834c169e30dced9219615655db17f5c95ceb1574a1987fda9603e57d6cb"}
Oct 30 21:50:09.325517 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.325315    4282 scope.go:122] "RemoveContainer" containerID="361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325"
Oct 30 21:50:09.332999 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.332974    4282 generic.go:356] "Generic (PLEG): container finished" podID="d0d4f9b0-d28f-4021-b9cd-e30732452bfc" containerID="4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93" exitCode=0
Oct 30 21:50:09.334606 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.333024    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"
Oct 30 21:50:09.334606 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.333044    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2" event={"ID":"d0d4f9b0-d28f-4021-b9cd-e30732452bfc","Type":"ContainerDied","Data":"4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93"}
Oct 30 21:50:09.334606 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.333332    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2" event={"ID":"d0d4f9b0-d28f-4021-b9cd-e30732452bfc","Type":"ContainerDied","Data":"16c9d407ae2383a6ca7ffa10cf933f01ea460eb529d14397ad0f179b60527b0d"}
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.336374    4282 generic.go:356] "Generic (PLEG): container finished" podID="da9755c2-d890-46e6-a83b-db0bd1ee9387" containerID="4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2" exitCode=0
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.336547    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j" event={"ID":"da9755c2-d890-46e6-a83b-db0bd1ee9387","Type":"ContainerDied","Data":"4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2"}
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.336590    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j" event={"ID":"da9755c2-d890-46e6-a83b-db0bd1ee9387","Type":"ContainerDied","Data":"a5c0e700988e33ee6264c876bbf66aedd4fe4c47192e9d05e99dfb11aae96370"}
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.336945    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.337431    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.337916    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.338654    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.338888    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.338972    4282 scope.go:122] "RemoveContainer" containerID="361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325"
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:50:09.339404    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325\": container with ID starting with 361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325 not found: ID does not exist" containerID="361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325"
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.339434    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325"} err="failed to get container status \"361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325\": rpc error: code = NotFound desc = could not find container \"361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325\": container with ID starting with 361b1a99b71a185533524514e06b99edd8d1674a5bbdc4711faa5f8738c4b325 not found: ID does not exist"
Oct 30 21:50:09.340732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.339455    4282 scope.go:122] "RemoveContainer" containerID="4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93"
Oct 30 21:50:09.351834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.351650    4282 scope.go:122] "RemoveContainer" containerID="4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93"
Oct 30 21:50:09.352278 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:50:09.351996    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93\": container with ID starting with 4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93 not found: ID does not exist" containerID="4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93"
Oct 30 21:50:09.352278 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.352031    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93"} err="failed to get container status \"4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93\": rpc error: code = NotFound desc = could not find container \"4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93\": container with ID starting with 4bad00b2b63ad077d87b9ddbc31241d207b093536010368ffd5d4396cd742d93 not found: ID does not exist"
Oct 30 21:50:09.352278 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.352052    4282 scope.go:122] "RemoveContainer" containerID="4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2"
Oct 30 21:50:09.356071 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.354181    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"]
Oct 30 21:50:09.356071 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.355969    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-681bcb24-597db44dff-5l5bs"]
Oct 30 21:50:09.362591 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.362570    4282 scope.go:122] "RemoveContainer" containerID="4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2"
Oct 30 21:50:09.362898 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:50:09.362877    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2\": container with ID starting with 4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2 not found: ID does not exist" containerID="4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2"
Oct 30 21:50:09.362965 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.362904    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2"} err="failed to get container status \"4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2\": rpc error: code = NotFound desc = could not find container \"4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2\": container with ID starting with 4c3263f30a8861d74d22f32e6d1cbdd26d657d4ab0afd899bf92c7a7133a20a2 not found: ID does not exist"
Oct 30 21:50:09.377297 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.377275    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"]
Oct 30 21:50:09.387673 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.387625    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4487d16a-5dd87f87bd-qzmk2"]
Oct 30 21:50:09.464164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.464132    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fb3c6f-e8fb-4cf6-8eaa-8241388d8161" path="/var/lib/kubelet/pods/71fb3c6f-e8fb-4cf6-8eaa-8241388d8161/volumes"
Oct 30 21:50:09.464577 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.464561    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d4f9b0-d28f-4021-b9cd-e30732452bfc" path="/var/lib/kubelet/pods/d0d4f9b0-d28f-4021-b9cd-e30732452bfc/volumes"
Oct 30 21:50:09.484592 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.484559    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"]
Oct 30 21:50:09.493181 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.493154    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-681bcb24-6885967fc5-6z45j"]
Oct 30 21:50:09.537315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.537288    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"]
Oct 30 21:50:09.539453 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:50:09.539427    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a025f32_89c7_4009_8b77_7ee839ed8c20.slice/crio-2e7be08d2995a5ce26b50c6740e6c7c0f7a187e649bc9f71b6ccf83f32aa760e WatchSource:0}: Error finding container 2e7be08d2995a5ce26b50c6740e6c7c0f7a187e649bc9f71b6ccf83f32aa760e: Status 404 returned error can't find the container with id 2e7be08d2995a5ce26b50c6740e6c7c0f7a187e649bc9f71b6ccf83f32aa760e
Oct 30 21:50:09.542448 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.542403    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:09.542556 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.542535    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:09.542647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:09.542589    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:10.258813 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.258775    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"]
Oct 30 21:50:10.262984 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.262963    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.270715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.270686    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0f26c5-openshift-default-dockercfg-tmpf5\""
Oct 30 21:50:10.277024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.276965    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"]
Oct 30 21:50:10.346711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.346679    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" event={"ID":"8a025f32-89c7-4009-8b77-7ee839ed8c20","Type":"ContainerStarted","Data":"f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec"}
Oct 30 21:50:10.346875 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.346733    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" event={"ID":"8a025f32-89c7-4009-8b77-7ee839ed8c20","Type":"ContainerStarted","Data":"2e7be08d2995a5ce26b50c6740e6c7c0f7a187e649bc9f71b6ccf83f32aa760e"}
Oct 30 21:50:10.373175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.373139    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.374779 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.373542    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.375074 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.375047    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wl59\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-kube-api-access-8wl59\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.375190 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.375174    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.375336 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.375311    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.375498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.375386    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.375498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.375437    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.375647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.375500    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.375647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.375547    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.437875 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.437809    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podStartSLOduration=2.437788302 podStartE2EDuration="2.437788302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:10.437684709 +0000 UTC m=+629.095780935" watchObservedRunningTime="2026-10-30 21:50:10.437788302 +0000 UTC m=+629.095884537"
Oct 30 21:50:10.476635 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.476594    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.476810    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.476889    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.476931    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.476977    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.477017    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.477071    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.477178    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.477234    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wl59\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-kube-api-access-8wl59\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.477263    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.477890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.477268    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.478187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.478163    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.478408 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.478383    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.478592 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.478570    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.479416 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.479388    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.480619 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.480598    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.491293 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.491265    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wl59\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-kube-api-access-8wl59\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.492246 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.492221    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.584928 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.584857    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:10.914419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.914330    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"]
Oct 30 21:50:10.917777 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:50:10.917749    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8d8505_4b73_4308_b7eb_4a778777e5d1.slice/crio-e43b356e3af5476b230c4274da2b7527b408843347611206b92510d0186439c4 WatchSource:0}: Error finding container e43b356e3af5476b230c4274da2b7527b408843347611206b92510d0186439c4: Status 404 returned error can't find the container with id e43b356e3af5476b230c4274da2b7527b408843347611206b92510d0186439c4
Oct 30 21:50:10.920117 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.920081    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:10.920211 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.920148    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:10.920211 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:10.920179    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:11.252175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.252141    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:11.257869 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.257843    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:11.354241 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.354201    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" event={"ID":"2a8d8505-4b73-4308-b7eb-4a778777e5d1","Type":"ContainerStarted","Data":"a6b8046ebdfbce6552775e7ed952654051fe5a5fdcb11f847e44edeca29e1ad0"}
Oct 30 21:50:11.354788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.354269    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" event={"ID":"2a8d8505-4b73-4308-b7eb-4a778777e5d1","Type":"ContainerStarted","Data":"e43b356e3af5476b230c4274da2b7527b408843347611206b92510d0186439c4"}
Oct 30 21:50:11.354788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.354712    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:11.355556 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.355531    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:11.415844 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.415799    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" podStartSLOduration=1.41578481 podStartE2EDuration="1.41578481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:11.412261604 +0000 UTC m=+630.070357827" watchObservedRunningTime="2026-10-30 21:50:11.41578481 +0000 UTC m=+630.073881032"
Oct 30 21:50:11.463017 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.462984    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9755c2-d890-46e6-a83b-db0bd1ee9387" path="/var/lib/kubelet/pods/da9755c2-d890-46e6-a83b-db0bd1ee9387/volumes"
Oct 30 21:50:11.585703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.585614    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:11.590712 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:11.590687    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:12.378030 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.378000    4282 generic.go:356] "Generic (PLEG): container finished" podID="70f33f6c-6ea7-4bde-8bb0-37c882ef7236" containerID="15e132574c6468202d8700fe872ecaa4dcf7c27897df85ac4a0464d753ebd28f" exitCode=0
Oct 30 21:50:12.378549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.378065    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc" event={"ID":"70f33f6c-6ea7-4bde-8bb0-37c882ef7236","Type":"ContainerDied","Data":"15e132574c6468202d8700fe872ecaa4dcf7c27897df85ac4a0464d753ebd28f"}
Oct 30 21:50:12.378549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.378453    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:12.379521 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.379501    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:12.390202 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.390184    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:50:12.498541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498514    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-certs\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.498705 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498591    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-kube-api-access-xszx4\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-kube-api-access-xszx4\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.498705 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498618    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-podinfo\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.498705 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498642    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-credential-socket\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.498705 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498673    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-token\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-token\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.498899 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498704    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-data\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.498899 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498782    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-envoy\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.498899 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498803    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-certs" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.498899 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498822    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-socket\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.498899 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498861    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istiod-ca-cert\") pod \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\" (UID: \"70f33f6c-6ea7-4bde-8bb0-37c882ef7236\") "
Oct 30 21:50:12.499153 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.498916    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-credential-socket" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.499153 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.499098    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-socket" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.499296 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.499271    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istiod-ca-cert" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:12.499760 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.499549    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-data" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.499887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.499805    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.499966 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.499923    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.499966 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.499940    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.499966 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.499956    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.501202 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.501180    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-envoy" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:50:12.501323 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.501304    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-podinfo" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:50:12.501617 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.501595    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-kube-api-access-xszx4" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "kube-api-access-xszx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:12.501719 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.501656    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-token" pod "70f33f6c-6ea7-4bde-8bb0-37c882ef7236" (UID: "70f33f6c-6ea7-4bde-8bb0-37c882ef7236"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:12.601694 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.601654    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xszx4\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-kube-api-access-xszx4\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.601694 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.601695    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.601886 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.601714    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.601886 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.601728    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.601886 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:12.601742    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f33f6c-6ea7-4bde-8bb0-37c882ef7236-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:13.384632 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:13.384589    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc" event={"ID":"70f33f6c-6ea7-4bde-8bb0-37c882ef7236","Type":"ContainerDied","Data":"dd0bb305f77de47bbe89d4f0beebdbdab4f6198ae518f9dbefaba667ca687d7c"}
Oct 30 21:50:13.385100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:13.384707    4282 scope.go:122] "RemoveContainer" containerID="15e132574c6468202d8700fe872ecaa4dcf7c27897df85ac4a0464d753ebd28f"
Oct 30 21:50:13.385100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:13.384942    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"
Oct 30 21:50:13.412829 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:13.412802    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"]
Oct 30 21:50:13.415493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:13.415451    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-31a18d9b-gw-openshift-default-55bfb7cd45-8lhdc"]
Oct 30 21:50:13.463609 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:13.463578    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f33f6c-6ea7-4bde-8bb0-37c882ef7236" path="/var/lib/kubelet/pods/70f33f6c-6ea7-4bde-8bb0-37c882ef7236/volumes"
Oct 30 21:50:15.189026 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.188991    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"]
Oct 30 21:50:15.189419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.189399    4282 cpu_manager.go:414] "containerMap: removing container" podUID="70f33f6c-6ea7-4bde-8bb0-37c882ef7236" containerName="istio-proxy"
Oct 30 21:50:15.189419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.189410    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f33f6c-6ea7-4bde-8bb0-37c882ef7236" containerName="istio-proxy"
Oct 30 21:50:15.189520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.189497    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f33f6c-6ea7-4bde-8bb0-37c882ef7236" containerName="istio-proxy"
Oct 30 21:50:15.194064 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.194049    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.200034 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.200014    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-2b2e72-serving-cert\""
Oct 30 21:50:15.217751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.217727    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"]
Oct 30 21:50:15.239646 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.239612    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.239781 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.239718    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.239852 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.239817    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26623cb1-5e21-4175-a760-60619be198cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.239914 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.239872    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6d6\" (UniqueName: \"kubernetes.io/projected/26623cb1-5e21-4175-a760-60619be198cf-kube-api-access-cn6d6\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.294294 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.294267    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:50:15.341260 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.341218    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26623cb1-5e21-4175-a760-60619be198cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.341384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.341260    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6d6\" (UniqueName: \"kubernetes.io/projected/26623cb1-5e21-4175-a760-60619be198cf-kube-api-access-cn6d6\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.341384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.341325    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.341384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.341360    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.342031 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.342005    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26623cb1-5e21-4175-a760-60619be198cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.344026 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.344007    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.344372 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.344352    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.374490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.373841    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6d6\" (UniqueName: \"kubernetes.io/projected/26623cb1-5e21-4175-a760-60619be198cf-kube-api-access-cn6d6\") pod \"maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.506563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.506451    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:15.796360 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.795245    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"]
Oct 30 21:50:15.798124 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:15.797390    4282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:50:16.433928 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:16.432432    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5" event={"ID":"26623cb1-5e21-4175-a760-60619be198cf","Type":"ContainerStarted","Data":"c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74"}
Oct 30 21:50:16.433928 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:16.432539    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5" event={"ID":"26623cb1-5e21-4175-a760-60619be198cf","Type":"ContainerStarted","Data":"8c63c64da2334119e019d94ab6013197a5553d3724b93b64d9f88bd1f10a7665"}
Oct 30 21:50:16.433928 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:16.432718    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:16.452846 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:16.452734    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5" podStartSLOduration=1.06113803 podStartE2EDuration="1.452719621s" totalImagesPullingTime="391.581591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:16.451704705 +0000 UTC m=+635.109800928" watchObservedRunningTime="2026-10-30 21:50:16.452719621 +0000 UTC m=+635.110815844"
Oct 30 21:50:18.502640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.502607    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"]
Oct 30 21:50:18.509743 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.509709    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.515280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.515260    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-0f26c5-serving-cert\""
Oct 30 21:50:18.520789 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.520768    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"]
Oct 30 21:50:18.596214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.596183    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-metrics-certs\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.596384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.596224    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.596384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.596257    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.596523 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.596396    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm2q7\" (UniqueName: \"kubernetes.io/projected/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-kube-api-access-xm2q7\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.698061 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.698023    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.698382 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.698360    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm2q7\" (UniqueName: \"kubernetes.io/projected/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-kube-api-access-xm2q7\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.698629 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.698612    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-metrics-certs\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.698775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.698760    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.713493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.701404    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.726754 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.722305    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm2q7\" (UniqueName: \"kubernetes.io/projected/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-kube-api-access-xm2q7\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.726754 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.725596    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-maas-api-tls\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.726754 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.726731    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-metrics-certs\") pod \"maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:18.826450 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:18.826115    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:19.138978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:19.138901    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"]
Oct 30 21:50:19.142499 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:50:19.142303    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a8d7f4_240c_4a9c_bdf8_b0bdd193d8cf.slice/crio-855c924a3a1a52108e6d2c6dbf2bb856dee02dcc8637ead415f77ce08eca444a WatchSource:0}: Error finding container 855c924a3a1a52108e6d2c6dbf2bb856dee02dcc8637ead415f77ce08eca444a: Status 404 returned error can't find the container with id 855c924a3a1a52108e6d2c6dbf2bb856dee02dcc8637ead415f77ce08eca444a
Oct 30 21:50:19.449068 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:19.449036    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7" event={"ID":"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf","Type":"ContainerStarted","Data":"855c924a3a1a52108e6d2c6dbf2bb856dee02dcc8637ead415f77ce08eca444a"}
Oct 30 21:50:20.472491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:20.471447    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7" event={"ID":"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf","Type":"ContainerStarted","Data":"f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66"}
Oct 30 21:50:20.472491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:20.472392    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:22.443129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:22.443102    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 21:50:22.482493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:22.476853    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7" podStartSLOduration=4.000624729 podStartE2EDuration="4.476832531s" totalImagesPullingTime="476.207802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:20.510632584 +0000 UTC m=+639.168728810" watchObservedRunningTime="2026-10-30 21:50:22.476832531 +0000 UTC m=+641.134928754"
Oct 30 21:50:23.601494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:23.593897    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:50:25.386493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:25.380662    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:50:25.386493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:25.380727    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:50:27.399916 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:27.399610    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:50:27.399916 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:27.399666    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:50:27.530754 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:27.530718    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 21:50:29.430166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:29.430138    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:50:30.610166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.610134    4282 generic.go:356] "Generic (PLEG): container finished" podID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerID="34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46" exitCode=0
Oct 30 21:50:30.612865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.610217    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" event={"ID":"4b416f91-5c95-43cd-bdc6-402bca298e02","Type":"ContainerDied","Data":"34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46"}
Oct 30 21:50:30.612865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.610719    4282 scope.go:122] "RemoveContainer" containerID="34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46"
Oct 30 21:50:30.612865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.612195    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:30.612865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.612258    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:30.612865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.612291    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:30.612865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.612737    4282 generic.go:356] "Generic (PLEG): container finished" podID="73d8a943-a45d-434a-a400-eec464164ea2" containerID="eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00" exitCode=0
Oct 30 21:50:30.612865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.612818    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" event={"ID":"73d8a943-a45d-434a-a400-eec464164ea2","Type":"ContainerDied","Data":"eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00"}
Oct 30 21:50:30.613693 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.613213    4282 scope.go:122] "RemoveContainer" containerID="eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00"
Oct 30 21:50:30.614602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.614509    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:30.614602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.614584    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:30.614676 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:30.614630    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:31.620305 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.620246    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" event={"ID":"4b416f91-5c95-43cd-bdc6-402bca298e02","Type":"ContainerStarted","Data":"7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c"}
Oct 30 21:50:31.622395 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.622369    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" event={"ID":"73d8a943-a45d-434a-a400-eec464164ea2","Type":"ContainerStarted","Data":"09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119"}
Oct 30 21:50:31.624070 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.624049    4282 generic.go:356] "Generic (PLEG): container finished" podID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerID="f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec" exitCode=0
Oct 30 21:50:31.624171 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.624111    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" event={"ID":"8a025f32-89c7-4009-8b77-7ee839ed8c20","Type":"ContainerDied","Data":"f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec"}
Oct 30 21:50:31.624514 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.624498    4282 scope.go:122] "RemoveContainer" containerID="f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec"
Oct 30 21:50:31.625794 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.625767    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:31.625881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.625833    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:31.625881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.625860    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:31.798516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.788916    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:31.798516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.789087    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:50:31.798516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:31.789143    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:50:32.206343 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:32.205904    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:32.206343 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:32.206031    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:50:32.206343 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:32.206073    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:50:32.641551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:32.639661    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" event={"ID":"8a025f32-89c7-4009-8b77-7ee839ed8c20","Type":"ContainerStarted","Data":"733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f"}
Oct 30 21:50:32.825512 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:32.790098    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:50:32.825512 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:32.790156    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:50:33.219232 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:33.217638    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:50:33.219232 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:33.217702    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:50:33.259795 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:33.259695    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:33.260852 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:33.260827    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:50:33.261179 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:33.261155    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:50:33.795495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:33.789004    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:50:33.795495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:33.789065    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:50:34.209494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:34.205856    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:50:34.209494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:34.205915    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:50:34.256494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:34.255890    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:50:34.256494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:34.255951    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:50:34.685503 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:50:34.685414    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:50:34.790276 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:34.789849    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:50:34.790276 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:34.789910    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:50:35.206298 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:35.205856    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:50:35.206298 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:35.205917    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:50:35.252109 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:35.251653    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:50:35.252109 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:35.251713    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:50:35.830528 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:35.789894    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:50:35.830528 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:35.789959    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:50:36.210305 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:36.209926    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:50:36.210305 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:36.209988    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:50:36.251449 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:36.251416    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:50:36.251667 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:36.251533    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:50:36.794340 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:36.793892    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 30 21:50:36.794340 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:36.793952    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 30 21:50:37.206172 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:37.206122    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 30 21:50:37.206455 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:37.206193    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 30 21:50:37.255873 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:37.255837    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:50:37.256862 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:37.256429    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 30 21:50:37.485988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:37.485898    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:50:37.791881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:37.791805    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:37.793999 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:37.792097    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:37.793999 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:37.792828    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 21:50:38.209764 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.209736    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:38.210073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.210051    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:38.210790 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.210771    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 21:50:38.256397 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.256369    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:38.257959 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.257936    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:38.258492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.258452    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 21:50:38.715027 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.714997    4282 generic.go:356] "Generic (PLEG): container finished" podID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerID="a6b8046ebdfbce6552775e7ed952654051fe5a5fdcb11f847e44edeca29e1ad0" exitCode=0
Oct 30 21:50:38.715215 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.715077    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" event={"ID":"2a8d8505-4b73-4308-b7eb-4a778777e5d1","Type":"ContainerDied","Data":"a6b8046ebdfbce6552775e7ed952654051fe5a5fdcb11f847e44edeca29e1ad0"}
Oct 30 21:50:38.715921 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.715902    4282 scope.go:122] "RemoveContainer" containerID="a6b8046ebdfbce6552775e7ed952654051fe5a5fdcb11f847e44edeca29e1ad0"
Oct 30 21:50:38.717028 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.717002    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:38.717084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.717075    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:38.717120 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:38.717109    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:50:39.722970 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:39.722927    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" event={"ID":"2a8d8505-4b73-4308-b7eb-4a778777e5d1","Type":"ContainerStarted","Data":"2b27b229f1a74197af93fbfebe56eb97366ae559ac88a1ab259af280eeddb65e"}
Oct 30 21:50:40.585873 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:40.585841    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:40.587039 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:40.587017    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:50:40.587207 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:40.587062    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:50:41.585872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:41.585833    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:50:41.586297 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:41.585907    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:50:42.379186 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.379149    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:42.478490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.478171    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"]
Oct 30 21:50:42.485415 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.485368    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.496676 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.494789    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"default-dockercfg-ck8pq\""
Oct 30 21:50:42.496676 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.494906    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"simulator-w1-0f26c5-kserve-self-signed-certs\""
Oct 30 21:50:42.496676 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.496020    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"kube-root-ca.crt\""
Oct 30 21:50:42.496676 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.496080    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"openshift-service-ca.crt\""
Oct 30 21:50:42.497687 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.497664    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"]
Oct 30 21:50:42.522904 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.522873    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-home\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.523014 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.522919    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tmp-dir\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.523094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.523070    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2kl\" (UniqueName: \"kubernetes.io/projected/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-kube-api-access-5t2kl\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.523152 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.523134    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-model-cache\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.523267 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.523248    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-dshm\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.523322 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.523308    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tls-certs\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.586264 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.586228    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:50:42.586700 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.586295    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:50:42.624697 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.624665    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-dshm\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.624870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.624707    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tls-certs\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.624870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.624749    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-home\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.624870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.624771    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tmp-dir\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.624870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.624825    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2kl\" (UniqueName: \"kubernetes.io/projected/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-kube-api-access-5t2kl\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.624870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.624854    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-model-cache\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.625197 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.625169    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tmp-dir\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.625276 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.625192    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-home\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.625276 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.625216    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-model-cache\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.629622 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.629561    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tls-certs\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.646461 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.646427    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-dshm\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.654509 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.650114    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2kl\" (UniqueName: \"kubernetes.io/projected/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-kube-api-access-5t2kl\") pod \"simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") " pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:42.812606 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:42.812572    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:43.112288 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:43.112265    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"]
Oct 30 21:50:43.114816 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:50:43.114783    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbedb77e_2f7a_4424_8e14_04e9f5b4cdfc.slice/crio-db8840cfad751a3aa2794d14ebd6b223f314b5768c58df19955ecd0b7b0698cb WatchSource:0}: Error finding container db8840cfad751a3aa2794d14ebd6b223f314b5768c58df19955ecd0b7b0698cb: Status 404 returned error can't find the container with id db8840cfad751a3aa2794d14ebd6b223f314b5768c58df19955ecd0b7b0698cb
Oct 30 21:50:43.585929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:43.585900    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:50:43.586081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:43.585952    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:50:43.750621 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:43.750582    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" event={"ID":"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc","Type":"ContainerStarted","Data":"6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1"}
Oct 30 21:50:43.750621 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:43.750628    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" event={"ID":"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc","Type":"ContainerStarted","Data":"db8840cfad751a3aa2794d14ebd6b223f314b5768c58df19955ecd0b7b0698cb"}
Oct 30 21:50:43.813258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:43.813174    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:43.814620 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:43.813829    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podStartSLOduration=1.813814126 podStartE2EDuration="1.813814126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:43.806170761 +0000 UTC m=+662.464266985" watchObservedRunningTime="2026-10-30 21:50:43.813814126 +0000 UTC m=+662.471910348"
Oct 30 21:50:44.585384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:44.585347    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:50:44.585640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:44.585413    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 21:50:45.589609 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:45.589583    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:45.590483 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:45.590444    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 21:50:46.608517 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:46.608485    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:50:52.813512 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:52.813451    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:52.825805 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:52.825786    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:53.813815 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:53.813783    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:50:57.475693 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:57.475666    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:50:58.795710 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.795682    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"]
Oct 30 21:50:58.799429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.799413    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:58.807543 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.807521    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"premium-w0-1529dc-kserve-self-signed-certs\""
Oct 30 21:50:58.810009 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.809990    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"default-dockercfg-w5jph\""
Oct 30 21:50:58.815799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.815780    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"openshift-service-ca.crt\""
Oct 30 21:50:58.815799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.815791    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"kube-root-ca.crt\""
Oct 30 21:50:58.819506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.819486    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"]
Oct 30 21:50:58.926244 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.926207    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-dshm\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:58.926610 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.926587    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-tmp-dir\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:58.929538 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.926737    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-model-cache\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:58.939342 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.929762    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqvp9\" (UniqueName: \"kubernetes.io/projected/a563a277-f347-4a67-b2a7-854282b5504a-kube-api-access-xqvp9\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:58.939656 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.939628    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-home\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:58.942240 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:58.942211    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a563a277-f347-4a67-b2a7-854282b5504a-tls-certs\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.043519 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043461    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-tmp-dir\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.043704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043525    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-model-cache\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.043704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043569    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqvp9\" (UniqueName: \"kubernetes.io/projected/a563a277-f347-4a67-b2a7-854282b5504a-kube-api-access-xqvp9\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.043704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043596    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-home\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.043704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043661    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a563a277-f347-4a67-b2a7-854282b5504a-tls-certs\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.043937 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043830    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-dshm\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.043937 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043853    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-tmp-dir\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.043937 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043912    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-model-cache\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.044089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.043948    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-home\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.046174 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.046120    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-dshm\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.046537 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.046517    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a563a277-f347-4a67-b2a7-854282b5504a-tls-certs\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.054775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.054752    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqvp9\" (UniqueName: \"kubernetes.io/projected/a563a277-f347-4a67-b2a7-854282b5504a-kube-api-access-xqvp9\") pod \"premium-w0-1529dc-kserve-76b65d7c68-wcc2q\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") " pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.111754 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.111736    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:50:59.395748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.395724    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"]
Oct 30 21:50:59.398127 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:50:59.398085    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda563a277_f347_4a67_b2a7_854282b5504a.slice/crio-6ae6c2b3d530244e9652b1e3726c850102f146caf48ae1938a70284a55686af1 WatchSource:0}: Error finding container 6ae6c2b3d530244e9652b1e3726c850102f146caf48ae1938a70284a55686af1: Status 404 returned error can't find the container with id 6ae6c2b3d530244e9652b1e3726c850102f146caf48ae1938a70284a55686af1
Oct 30 21:50:59.844147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.844110    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" event={"ID":"a563a277-f347-4a67-b2a7-854282b5504a","Type":"ContainerStarted","Data":"ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d"}
Oct 30 21:50:59.844572 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.844165    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" event={"ID":"a563a277-f347-4a67-b2a7-854282b5504a","Type":"ContainerStarted","Data":"6ae6c2b3d530244e9652b1e3726c850102f146caf48ae1938a70284a55686af1"}
Oct 30 21:50:59.864349 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:50:59.864302    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podStartSLOduration=1.864288858 podStartE2EDuration="1.864288858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:59.861955078 +0000 UTC m=+678.520051301" watchObservedRunningTime="2026-10-30 21:50:59.864288858 +0000 UTC m=+678.522385081"
Oct 30 21:51:00.112677 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:00.112584    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:51:04.654847 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:51:04.654805    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:51:09.112289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:09.112253    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:51:09.124857 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:09.124831    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:51:09.182893 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:09.182868    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:51:09.909073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:09.909042    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:51:22.305649 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.305616    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"]
Oct 30 21:51:22.310719 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.310701    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.318927 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.318901    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b2e72-openshift-default-dockercfg-98gbf\""
Oct 30 21:51:22.330213 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.330188    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"]
Oct 30 21:51:22.429567 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.429535    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.429757 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.429587    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.429757 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.429610    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.429757 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.429693    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.429757 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.429719    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.429981 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.429868    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.429981 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.429933    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqd8\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-kube-api-access-hjqd8\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.429981 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.429964    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f80220f4-6a6b-417e-ae8d-e373627a5425-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.430129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.430008    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.530794 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.530767    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.530961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.530810    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.530961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.530836    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.530961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.530864    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.530961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.530957    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531005    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqd8\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-kube-api-access-hjqd8\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531077    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f80220f4-6a6b-417e-ae8d-e373627a5425-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531127    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531180    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531194    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531236    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531302    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531519 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531498    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.531732 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.531712    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f80220f4-6a6b-417e-ae8d-e373627a5425-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.533710 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.533682    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.533815 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.533799    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.544967 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.544948    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.547521 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.547502    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqd8\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-kube-api-access-hjqd8\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.624367 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.624309    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:22.913119 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.913048    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"]
Oct 30 21:51:22.916391 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:51:22.916362    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80220f4_6a6b_417e_ae8d_e373627a5425.slice/crio-37a7e36689ccfca3be2b935b2c915ff45692545bdedf748df7c07f710afc0df0 WatchSource:0}: Error finding container 37a7e36689ccfca3be2b935b2c915ff45692545bdedf748df7c07f710afc0df0: Status 404 returned error can't find the container with id 37a7e36689ccfca3be2b935b2c915ff45692545bdedf748df7c07f710afc0df0
Oct 30 21:51:22.918630 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.918596    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:51:22.918692 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.918664    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:51:22.918728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.918692    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:51:22.986671 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.986643    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" event={"ID":"f80220f4-6a6b-417e-ae8d-e373627a5425","Type":"ContainerStarted","Data":"56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237"}
Oct 30 21:51:22.986807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:22.986692    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" event={"ID":"f80220f4-6a6b-417e-ae8d-e373627a5425","Type":"ContainerStarted","Data":"37a7e36689ccfca3be2b935b2c915ff45692545bdedf748df7c07f710afc0df0"}
Oct 30 21:51:23.013118 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:23.013060    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podStartSLOduration=1.013041517 podStartE2EDuration="1.013041517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:23.009728621 +0000 UTC m=+701.667824845" watchObservedRunningTime="2026-10-30 21:51:23.013041517 +0000 UTC m=+701.671137740"
Oct 30 21:51:23.625328 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:23.625297    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:23.626895 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:23.626871    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:51:23.627022 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:23.626916    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 21:51:24.625447 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:24.625409    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:51:24.625884 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:24.625490    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 21:51:25.624797 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:25.624765    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:51:25.624973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:25.624829    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 21:51:26.625428 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:26.625394    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:51:26.625878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:26.625441    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 21:51:27.625439 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:27.625399    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:51:27.625855 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:27.625458    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 21:51:28.625237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:28.625197    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Oct 30 21:51:28.625444 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:28.625254    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Oct 30 21:51:29.307795 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:29.307762    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:51:29.629412 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:29.629341    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:29.629707 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:29.629687    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:29.630345 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:29.630329    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 21:51:34.650354 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:51:34.650315    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:51:59.004771 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:51:59.004738    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:52:04.670097 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:52:04.670060    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:52:17.099180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:17.099149    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:52:22.327692 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.327642    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"]
Oct 30 21:52:22.347291 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.347247    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.370428 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.370403    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:52:22.370577 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.370504    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:52:22.370577 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.370520    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:52:22.378795 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.378774    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"]
Oct 30 21:52:22.383101 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.383079    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bb78d58d0bb0\""
Oct 30 21:52:22.383985 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.383964    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:52:22.384113 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.384081    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-tvdlq\""
Oct 30 21:52:22.403792 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.403761    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-provider-credentials\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.403918 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.403814    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-plugins-config-volume\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.403918 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.403909    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnq2g\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-kube-api-access-dnq2g\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.404004 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.403940    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-routing-overlay-volume\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.505323 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.505293    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-provider-credentials\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.505529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.505342    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-plugins-config-volume\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.505529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.505379    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnq2g\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-kube-api-access-dnq2g\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.505673 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.505547    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-routing-overlay-volume\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.506196 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.506110    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-routing-overlay-volume\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.506196 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.506146    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-plugins-config-volume\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.507995 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.507968    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-provider-credentials\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.526205 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.526183    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnq2g\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-kube-api-access-dnq2g\") pod \"payload-processing-external-model-5768c7cdbc-476l8\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") " pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.660645 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.660562    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:22.953784 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:22.953760    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"]
Oct 30 21:52:22.956161 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:52:22.956133    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a742855_97dc_43c2_802b_4bc268997f67.slice/crio-6846296e993071feb3d14243459fe8c0f56adebb9d523b907dfb939a3848f213 WatchSource:0}: Error finding container 6846296e993071feb3d14243459fe8c0f56adebb9d523b907dfb939a3848f213: Status 404 returned error can't find the container with id 6846296e993071feb3d14243459fe8c0f56adebb9d523b907dfb939a3848f213
Oct 30 21:52:23.344358 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.344281    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"]
Oct 30 21:52:23.348080 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.348055    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.365021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.365000    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"]
Oct 30 21:52:23.366094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.366059    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"distinct-w0-1529dc-kserve-self-signed-certs\""
Oct 30 21:52:23.417198 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.417174    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkd4p\" (UniqueName: \"kubernetes.io/projected/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-kube-api-access-fkd4p\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.417329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.417210    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tls-certs\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.417329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.417299    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-dshm\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.417407 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.417342    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tmp-dir\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.417407 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.417398    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-home\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.417531 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.417516    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-model-cache\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.516187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.516151    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8" event={"ID":"4a742855-97dc-43c2-802b-4bc268997f67","Type":"ContainerStarted","Data":"346ca2d3e2820fe7805dacb48b61e4e4c670bd33444e72f0bdf0dce953c5a68a"}
Oct 30 21:52:23.516447 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.516431    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8" event={"ID":"4a742855-97dc-43c2-802b-4bc268997f67","Type":"ContainerStarted","Data":"6846296e993071feb3d14243459fe8c0f56adebb9d523b907dfb939a3848f213"}
Oct 30 21:52:23.518043 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.518024    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:23.519651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.519623    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkd4p\" (UniqueName: \"kubernetes.io/projected/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-kube-api-access-fkd4p\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.519825 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.519805    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tls-certs\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.522990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.521739    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-dshm\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.522990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.521776    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tmp-dir\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.522990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.521833    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-home\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.522990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.521959    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-model-cache\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.522990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.522311    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-model-cache\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.522990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.522678    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tmp-dir\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.522990 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.522699    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-home\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.525643 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.525614    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tls-certs\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.526166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.526142    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-dshm\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.532972 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.532944    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkd4p\" (UniqueName: \"kubernetes.io/projected/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-kube-api-access-fkd4p\") pod \"distinct-w0-1529dc-kserve-84778c9587-rt427\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.540568 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.540520    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8" podStartSLOduration=1.5405069980000001 podStartE2EDuration="1.540506998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:23.538099565 +0000 UTC m=+762.196195788" watchObservedRunningTime="2026-10-30 21:52:23.540506998 +0000 UTC m=+762.198603448"
Oct 30 21:52:23.660599 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.660527    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:23.943729 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:23.943634    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"]
Oct 30 21:52:23.945811 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:52:23.945776    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae7e3af0_2a59_4f1b_bc9c_7530aec5ae75.slice/crio-9308e94ffa6c9028da0a1afeab2efa8b06a2f02a56cd29b3461be05f3fb3b6fd WatchSource:0}: Error finding container 9308e94ffa6c9028da0a1afeab2efa8b06a2f02a56cd29b3461be05f3fb3b6fd: Status 404 returned error can't find the container with id 9308e94ffa6c9028da0a1afeab2efa8b06a2f02a56cd29b3461be05f3fb3b6fd
Oct 30 21:52:24.324954 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.324871    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"]
Oct 30 21:52:24.328594 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.328571    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.344801 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.344775    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"]
Oct 30 21:52:24.447819 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.447790    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.447819 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.447822    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.448021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.447851    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.448021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.447906    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6jp\" (UniqueName: \"kubernetes.io/projected/f536d905-2529-4495-80ff-1d16c5e2cc8a-kube-api-access-9f6jp\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.448021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.447958    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.448021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.448005    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.448150 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.448082    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.448150 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.448123    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.448212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.448154    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f536d905-2529-4495-80ff-1d16c5e2cc8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.522888 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.522848    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" event={"ID":"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75","Type":"ContainerStarted","Data":"948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050"}
Oct 30 21:52:24.523038 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.522897    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" event={"ID":"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75","Type":"ContainerStarted","Data":"9308e94ffa6c9028da0a1afeab2efa8b06a2f02a56cd29b3461be05f3fb3b6fd"}
Oct 30 21:52:24.542723 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.542669    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podStartSLOduration=1.542651782 podStartE2EDuration="1.542651782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:24.541284365 +0000 UTC m=+763.199380601" watchObservedRunningTime="2026-10-30 21:52:24.542651782 +0000 UTC m=+763.200748006"
Oct 30 21:52:24.549775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.549750    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.549905 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.549784    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.549970 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.549906    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.549970 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.549937    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6jp\" (UniqueName: \"kubernetes.io/projected/f536d905-2529-4495-80ff-1d16c5e2cc8a-kube-api-access-9f6jp\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550121 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550097    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550149    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550188    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550373 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550238    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550373 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550234    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550373 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550277    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f536d905-2529-4495-80ff-1d16c5e2cc8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550636 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550424    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550636 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550574    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.550980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.550960    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f536d905-2529-4495-80ff-1d16c5e2cc8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.551076 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.551059    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.552143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.552124    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.552590 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.552574    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.562831 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.562808    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f536d905-2529-4495-80ff-1d16c5e2cc8a-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.568711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.568691    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6jp\" (UniqueName: \"kubernetes.io/projected/f536d905-2529-4495-80ff-1d16c5e2cc8a-kube-api-access-9f6jp\") pod \"maas-default-gateway-openshift-default-85544d4cd7-c5n7j\" (UID: \"f536d905-2529-4495-80ff-1d16c5e2cc8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.641534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.641458    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:24.668510 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.664343    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:24.975892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.975867    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"]
Oct 30 21:52:24.976856 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:52:24.976836    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf536d905_2529_4495_80ff_1d16c5e2cc8a.slice/crio-5d7f82c8429a34ae5c96ee6e202a81bdbb20c3ae5b42c135406f9d837cd73dcb WatchSource:0}: Error finding container 5d7f82c8429a34ae5c96ee6e202a81bdbb20c3ae5b42c135406f9d837cd73dcb: Status 404 returned error can't find the container with id 5d7f82c8429a34ae5c96ee6e202a81bdbb20c3ae5b42c135406f9d837cd73dcb
Oct 30 21:52:24.979162 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.979126    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:52:24.979266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.979198    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:52:24.979266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:24.979229    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:52:25.529299 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:25.529264    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" event={"ID":"f536d905-2529-4495-80ff-1d16c5e2cc8a","Type":"ContainerStarted","Data":"a27796c504f58146cf5e997c7add83b16cb6904039fad1fc52fd494951009bbf"}
Oct 30 21:52:25.529836 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:25.529327    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" event={"ID":"f536d905-2529-4495-80ff-1d16c5e2cc8a","Type":"ContainerStarted","Data":"5d7f82c8429a34ae5c96ee6e202a81bdbb20c3ae5b42c135406f9d837cd73dcb"}
Oct 30 21:52:25.576870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:25.576802    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podStartSLOduration=1.5767833759999998 podStartE2EDuration="1.576783376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:25.57496886 +0000 UTC m=+764.233065096" watchObservedRunningTime="2026-10-30 21:52:25.576783376 +0000 UTC m=+764.234879600"
Oct 30 21:52:25.642549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:25.642522    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:26.643533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:26.643502    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:52:26.643935 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:26.643548    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 21:52:27.642320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:27.642285    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:52:27.642532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:27.642337    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 21:52:28.641966 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:28.641934    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:52:28.642364 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:28.641987    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 21:52:29.506866 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.506837    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"]
Oct 30 21:52:29.509723 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.509700    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:29.550300 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.550274    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"]
Oct 30 21:52:29.586751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.586716    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.591284 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.591254    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c709a0b1f5bb\""
Oct 30 21:52:29.596793 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.596375    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"]
Oct 30 21:52:29.652499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.639926    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-plugins-config-volume\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.662506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.653214    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-routing-overlay-volume\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.662812 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.662786    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6v9n\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-kube-api-access-r6v9n\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.662967 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.662952    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-provider-credentials\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.663166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.643860    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:52:29.678545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.663301    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 21:52:29.768229 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.768132    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-plugins-config-volume\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.768546 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.768524    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-routing-overlay-volume\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.768712 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.768695    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6v9n\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-kube-api-access-r6v9n\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.768844 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.768828    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-provider-credentials\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.770188 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.770162    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-routing-overlay-volume\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.771390 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.771326    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-plugins-config-volume\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.783278 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.783251    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-provider-credentials\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.784853 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.784830    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6v9n\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-kube-api-access-r6v9n\") pod \"payload-processing-external-model-6bd796bf-8fpp6\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:29.903357 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:29.903327    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:30.241260 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:30.240978    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"]
Oct 30 21:52:30.244414 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:52:30.244229    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f66bbe_3004_488f_a016_601b8bb54da4.slice/crio-205685d2b79a88a9927923e2e4a8263cb87dcfe509556f25cbe791a212935736 WatchSource:0}: Error finding container 205685d2b79a88a9927923e2e4a8263cb87dcfe509556f25cbe791a212935736: Status 404 returned error can't find the container with id 205685d2b79a88a9927923e2e4a8263cb87dcfe509556f25cbe791a212935736
Oct 30 21:52:30.608648 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:30.608503    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6" event={"ID":"11f66bbe-3004-488f-a016-601b8bb54da4","Type":"ContainerStarted","Data":"e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee"}
Oct 30 21:52:30.608648 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:30.608572    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6" event={"ID":"11f66bbe-3004-488f-a016-601b8bb54da4","Type":"ContainerStarted","Data":"205685d2b79a88a9927923e2e4a8263cb87dcfe509556f25cbe791a212935736"}
Oct 30 21:52:30.610845 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:30.610285    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:30.631243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:30.631169    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6" podStartSLOduration=1.6311529729999998 podStartE2EDuration="1.631152973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:30.628895889 +0000 UTC m=+769.286992113" watchObservedRunningTime="2026-10-30 21:52:30.631152973 +0000 UTC m=+769.289249198"
Oct 30 21:52:30.645955 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:30.645920    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:52:30.646089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:30.645978    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 21:52:31.642376 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:31.642343    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:52:31.643016 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:31.642985    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 21:52:32.642122 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:32.642097    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 30 21:52:32.642332 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:32.642314    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 30 21:52:33.661505 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:33.661451    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:33.677182 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:33.677160    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:34.309568 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.309532    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:34.310297 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.310274    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:34.311084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.311062    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j"
Oct 30 21:52:34.507643 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.507586    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8" podUID="4a742855-97dc-43c2-802b-4bc268997f67" containerName="payload-processing" containerID="cri-o://346ca2d3e2820fe7805dacb48b61e4e4c670bd33444e72f0bdf0dce953c5a68a" gracePeriod=25
Oct 30 21:52:34.653622 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.653589    4282 generic.go:356] "Generic (PLEG): container finished" podID="4a742855-97dc-43c2-802b-4bc268997f67" containerID="346ca2d3e2820fe7805dacb48b61e4e4c670bd33444e72f0bdf0dce953c5a68a" exitCode=0
Oct 30 21:52:34.654254 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.654234    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8" event={"ID":"4a742855-97dc-43c2-802b-4bc268997f67","Type":"ContainerDied","Data":"346ca2d3e2820fe7805dacb48b61e4e4c670bd33444e72f0bdf0dce953c5a68a"}
Oct 30 21:52:34.654650 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:52:34.654621    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:52:34.666505 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.666435    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:52:34.796497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.795994    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:34.874698 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.874665    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-kube-api-access-dnq2g\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-kube-api-access-dnq2g\") pod \"4a742855-97dc-43c2-802b-4bc268997f67\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") "
Oct 30 21:52:34.874872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.874796    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-provider-credentials\") pod \"4a742855-97dc-43c2-802b-4bc268997f67\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") "
Oct 30 21:52:34.874872 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.874851    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-routing-overlay-volume\") pod \"4a742855-97dc-43c2-802b-4bc268997f67\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") "
Oct 30 21:52:34.874993 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.874896    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-plugins-config-volume\") pod \"4a742855-97dc-43c2-802b-4bc268997f67\" (UID: \"4a742855-97dc-43c2-802b-4bc268997f67\") "
Oct 30 21:52:34.875297 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.875266    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-routing-overlay-volume" pod "4a742855-97dc-43c2-802b-4bc268997f67" (UID: "4a742855-97dc-43c2-802b-4bc268997f67"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:34.875484 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.875369    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-plugins-config-volume" pod "4a742855-97dc-43c2-802b-4bc268997f67" (UID: "4a742855-97dc-43c2-802b-4bc268997f67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:34.877147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.877091    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-kube-api-access-dnq2g" pod "4a742855-97dc-43c2-802b-4bc268997f67" (UID: "4a742855-97dc-43c2-802b-4bc268997f67"). InnerVolumeSpecName "kube-api-access-dnq2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:34.877230 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.877159    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-provider-credentials" pod "4a742855-97dc-43c2-802b-4bc268997f67" (UID: "4a742855-97dc-43c2-802b-4bc268997f67"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:34.976195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.976160    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnq2g\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-kube-api-access-dnq2g\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:34.976195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.976190    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a742855-97dc-43c2-802b-4bc268997f67-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:34.976195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.976200    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:34.976422 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:34.976209    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a742855-97dc-43c2-802b-4bc268997f67-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:35.664133 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:35.664087    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8" event={"ID":"4a742855-97dc-43c2-802b-4bc268997f67","Type":"ContainerDied","Data":"6846296e993071feb3d14243459fe8c0f56adebb9d523b907dfb939a3848f213"}
Oct 30 21:52:35.664328 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:35.664173    4282 scope.go:122] "RemoveContainer" containerID="346ca2d3e2820fe7805dacb48b61e4e4c670bd33444e72f0bdf0dce953c5a68a"
Oct 30 21:52:35.664414 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:35.664345    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"
Oct 30 21:52:35.687835 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:35.687812    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"]
Oct 30 21:52:35.691429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:35.691400    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5768c7cdbc-476l8"]
Oct 30 21:52:37.463642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:37.463610    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a742855-97dc-43c2-802b-4bc268997f67" path="/var/lib/kubelet/pods/4a742855-97dc-43c2-802b-4bc268997f67/volumes"
Oct 30 21:52:41.733175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:41.733143    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:52:42.039207 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.039170    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"]
Oct 30 21:52:42.043279 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.043246    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:42.078637 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.078608    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"]
Oct 30 21:52:42.079191 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.079164    4282 cpu_manager.go:414] "containerMap: removing container" podUID="4a742855-97dc-43c2-802b-4bc268997f67" containerName="payload-processing"
Oct 30 21:52:42.079191 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.079183    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a742855-97dc-43c2-802b-4bc268997f67" containerName="payload-processing"
Oct 30 21:52:42.079450 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.079298    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a742855-97dc-43c2-802b-4bc268997f67" containerName="payload-processing"
Oct 30 21:52:42.083897 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.083879    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.096326 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.096308    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fdda022c1562\""
Oct 30 21:52:42.096414 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.096386    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-89a8db81fd74\""
Oct 30 21:52:42.103333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.103309    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"]
Oct 30 21:52:42.279585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.279553    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-routing-overlay-volume\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.279759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.279599    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgk7f\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-kube-api-access-fgk7f\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.279759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.279678    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-provider-credentials\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.279834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.279806    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-plugins-config-volume\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.380633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.380552    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgk7f\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-kube-api-access-fgk7f\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.380633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.380604    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-provider-credentials\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.380833 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.380682    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-plugins-config-volume\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.380833 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.380740    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-routing-overlay-volume\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.381643 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.381615    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-routing-overlay-volume\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.381735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.381668    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-plugins-config-volume\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.383394 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.383376    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-provider-credentials\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.392782 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.392758    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgk7f\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-kube-api-access-fgk7f\") pod \"payload-processing-external-model-5c748545bd-msq2t\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") " pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.396376 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.396353    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:42.680975 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:42.680931    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"]
Oct 30 21:52:43.797494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:43.796500    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t" event={"ID":"b864bc71-aa90-44b1-88ca-b5634874544d","Type":"ContainerStarted","Data":"3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971"}
Oct 30 21:52:43.797494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:43.796566    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t" event={"ID":"b864bc71-aa90-44b1-88ca-b5634874544d","Type":"ContainerStarted","Data":"09b9a2827fc542b2503c4657a10125d2acafd1414110dec9901cd419fd8a24d7"}
Oct 30 21:52:43.797494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:43.797484    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:43.834316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:43.821604    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t" podStartSLOduration=1.821586165 podStartE2EDuration="1.821586165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:43.818890443 +0000 UTC m=+782.476986693" watchObservedRunningTime="2026-10-30 21:52:43.821586165 +0000 UTC m=+782.479682400"
Oct 30 21:52:47.040022 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.039964    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6" podUID="11f66bbe-3004-488f-a016-601b8bb54da4" containerName="payload-processing" containerID="cri-o://e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee" gracePeriod=25
Oct 30 21:52:47.300558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.300537    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:47.381357 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.381327    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-routing-overlay-volume\") pod \"11f66bbe-3004-488f-a016-601b8bb54da4\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") "
Oct 30 21:52:47.381517 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.381435    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-provider-credentials\") pod \"11f66bbe-3004-488f-a016-601b8bb54da4\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") "
Oct 30 21:52:47.381604 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.381517    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-kube-api-access-r6v9n\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-kube-api-access-r6v9n\") pod \"11f66bbe-3004-488f-a016-601b8bb54da4\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") "
Oct 30 21:52:47.381604 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.381564    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-plugins-config-volume\") pod \"11f66bbe-3004-488f-a016-601b8bb54da4\" (UID: \"11f66bbe-3004-488f-a016-601b8bb54da4\") "
Oct 30 21:52:47.381713 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.381685    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-routing-overlay-volume" pod "11f66bbe-3004-488f-a016-601b8bb54da4" (UID: "11f66bbe-3004-488f-a016-601b8bb54da4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:47.381983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.381950    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-plugins-config-volume" pod "11f66bbe-3004-488f-a016-601b8bb54da4" (UID: "11f66bbe-3004-488f-a016-601b8bb54da4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:47.381983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.381977    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:47.383731 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.383711    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-kube-api-access-r6v9n" pod "11f66bbe-3004-488f-a016-601b8bb54da4" (UID: "11f66bbe-3004-488f-a016-601b8bb54da4"). InnerVolumeSpecName "kube-api-access-r6v9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:47.383809 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.383730    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-provider-credentials" pod "11f66bbe-3004-488f-a016-601b8bb54da4" (UID: "11f66bbe-3004-488f-a016-601b8bb54da4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:47.482893 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.482864    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:47.482893 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.482887    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6v9n\" (UniqueName: \"kubernetes.io/projected/11f66bbe-3004-488f-a016-601b8bb54da4-kube-api-access-r6v9n\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:47.482893 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.482896    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11f66bbe-3004-488f-a016-601b8bb54da4-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:52:47.858853 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.858822    4282 generic.go:356] "Generic (PLEG): container finished" podID="11f66bbe-3004-488f-a016-601b8bb54da4" containerID="e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee" exitCode=0
Oct 30 21:52:47.859055 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.858871    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"
Oct 30 21:52:47.859055 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.858915    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6" event={"ID":"11f66bbe-3004-488f-a016-601b8bb54da4","Type":"ContainerDied","Data":"e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee"}
Oct 30 21:52:47.859055 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.858976    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6" event={"ID":"11f66bbe-3004-488f-a016-601b8bb54da4","Type":"ContainerDied","Data":"205685d2b79a88a9927923e2e4a8263cb87dcfe509556f25cbe791a212935736"}
Oct 30 21:52:47.859055 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.859006    4282 scope.go:122] "RemoveContainer" containerID="e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee"
Oct 30 21:52:47.867970 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.867955    4282 scope.go:122] "RemoveContainer" containerID="e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee"
Oct 30 21:52:47.868199 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:52:47.868182    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee\": container with ID starting with e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee not found: ID does not exist" containerID="e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee"
Oct 30 21:52:47.868242 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.868209    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee"} err="failed to get container status \"e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee\": rpc error: code = NotFound desc = could not find container \"e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee\": container with ID starting with e6e8b209017d9824762013623e3655661ace534daea47e0a7166f2d1f4dea2ee not found: ID does not exist"
Oct 30 21:52:47.874024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.873999    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"]
Oct 30 21:52:47.879177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:47.879160    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-8fpp6"]
Oct 30 21:52:49.462867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:49.462836    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f66bbe-3004-488f-a016-601b8bb54da4" path="/var/lib/kubelet/pods/11f66bbe-3004-488f-a016-601b8bb54da4/volumes"
Oct 30 21:52:55.852995 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:55.852964    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:52:56.728498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.728019    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"]
Oct 30 21:52:56.737491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.728751    4282 cpu_manager.go:414] "containerMap: removing container" podUID="11f66bbe-3004-488f-a016-601b8bb54da4" containerName="payload-processing"
Oct 30 21:52:56.737491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.728774    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f66bbe-3004-488f-a016-601b8bb54da4" containerName="payload-processing"
Oct 30 21:52:56.737491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.728894    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f66bbe-3004-488f-a016-601b8bb54da4" containerName="payload-processing"
Oct 30 21:52:56.831398 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.831359    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:56.838493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.833429    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"]
Oct 30 21:52:56.879499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.877621    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"kube-root-ca.crt\""
Oct 30 21:52:56.893496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.891609    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"openshift-service-ca.crt\""
Oct 30 21:52:56.893496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.891746    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"default-dockercfg-tsrf9\""
Oct 30 21:52:56.893496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.891891    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"distinct-2-w4-939a16-kserve-self-signed-certs\""
Oct 30 21:52:56.946495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.943766    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-tmp-dir\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:56.946495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.943909    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-model-cache\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:56.946495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.943945    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-home\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:56.946495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.943991    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-dshm\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:56.946495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.944028    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfchk\" (UniqueName: \"kubernetes.io/projected/16c6d10d-2eea-4866-a833-59cb87ba67e3-kube-api-access-rfchk\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:56.946495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:56.944109    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c6d10d-2eea-4866-a833-59cb87ba67e3-tls-certs\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.053493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.049100    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-tmp-dir\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.069399 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.053812    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-model-cache\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.069660 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.069634    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-home\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.069832 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.069814    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-dshm\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.069970 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.069951    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfchk\" (UniqueName: \"kubernetes.io/projected/16c6d10d-2eea-4866-a833-59cb87ba67e3-kube-api-access-rfchk\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.070144 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.070128    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c6d10d-2eea-4866-a833-59cb87ba67e3-tls-certs\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.071313 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.050258    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-tmp-dir\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.071313 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.059777    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-model-cache\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.071313 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.071031    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-home\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.117897 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.117826    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-dshm\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.120107 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.120082    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c6d10d-2eea-4866-a833-59cb87ba67e3-tls-certs\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.137248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.137225    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfchk\" (UniqueName: \"kubernetes.io/projected/16c6d10d-2eea-4866-a833-59cb87ba67e3-kube-api-access-rfchk\") pod \"distinct-2-w4-939a16-kserve-866f847b7-q65jj\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:57.334773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:57.334677    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:52:58.348359 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:58.333413    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"]
Oct 30 21:52:59.143068 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:59.143032    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" event={"ID":"16c6d10d-2eea-4866-a833-59cb87ba67e3","Type":"ContainerStarted","Data":"ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04"}
Oct 30 21:52:59.143243 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:59.143082    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" event={"ID":"16c6d10d-2eea-4866-a833-59cb87ba67e3","Type":"ContainerStarted","Data":"92869d88daa0394c08e7ef0d683a0c693396f959e2153c375ff3d9b8425b2434"}
Oct 30 21:52:59.207637 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:59.207587    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podStartSLOduration=3.207573675 podStartE2EDuration="3.207573675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:59.20622141 +0000 UTC m=+797.864317632" watchObservedRunningTime="2026-10-30 21:52:59.207573675 +0000 UTC m=+797.865669898"
Oct 30 21:52:59.335688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:52:59.335654    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:53:04.649809 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:53:04.649769    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:53:06.134248 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:06.134215    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:53:06.956206 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:06.956164    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"]
Oct 30 21:53:06.959931 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:06.959909    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:06.968187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:06.968157    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"default-dockercfg-86s4f\""
Oct 30 21:53:06.968187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:06.968165    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"distinct-w3-ceb36a-kserve-self-signed-certs\""
Oct 30 21:53:06.974160 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:06.974141    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"openshift-service-ca.crt\""
Oct 30 21:53:06.974373 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:06.974358    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"kube-root-ca.crt\""
Oct 30 21:53:06.974600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:06.974579    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"]
Oct 30 21:53:07.026833 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.026799    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-dshm\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.026966 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.026845    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tscn2\" (UniqueName: \"kubernetes.io/projected/deecd927-414a-41ec-9341-89cc0af48c7f-kube-api-access-tscn2\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.026966 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.026867    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-tmp-dir\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.027094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.026983    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deecd927-414a-41ec-9341-89cc0af48c7f-tls-certs\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.027094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.027033    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-model-cache\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.027172 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.027111    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-home\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.129235 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.129186    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-dshm\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.129532 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.129511    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tscn2\" (UniqueName: \"kubernetes.io/projected/deecd927-414a-41ec-9341-89cc0af48c7f-kube-api-access-tscn2\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.129654 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.129640    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-tmp-dir\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.129740 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.129729    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deecd927-414a-41ec-9341-89cc0af48c7f-tls-certs\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.129833 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.129819    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-model-cache\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.129953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.129942    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-home\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.130500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.130457    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-home\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.134837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.134812    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-tmp-dir\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.148632 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.138773    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-model-cache\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.149069 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.149046    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-dshm\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.163346 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.163310    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tscn2\" (UniqueName: \"kubernetes.io/projected/deecd927-414a-41ec-9341-89cc0af48c7f-kube-api-access-tscn2\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.183607 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.182174    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deecd927-414a-41ec-9341-89cc0af48c7f-tls-certs\") pod \"distinct-w3-ceb36a-kserve-56fc566456-8k4mb\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.273539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.273338    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:07.336104 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.336074    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:53:07.357863 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.357609    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:53:07.590762 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:07.590528    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"]
Oct 30 21:53:07.593604 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:53:07.593575    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeecd927_414a_41ec_9341_89cc0af48c7f.slice/crio-626e97090ee253247a427f9275922b416f5ef1d35712f1216b66f67de16e551e WatchSource:0}: Error finding container 626e97090ee253247a427f9275922b416f5ef1d35712f1216b66f67de16e551e: Status 404 returned error can't find the container with id 626e97090ee253247a427f9275922b416f5ef1d35712f1216b66f67de16e551e
Oct 30 21:53:08.214178 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:08.214142    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" event={"ID":"deecd927-414a-41ec-9341-89cc0af48c7f","Type":"ContainerStarted","Data":"38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6"}
Oct 30 21:53:08.214727 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:08.214195    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" event={"ID":"deecd927-414a-41ec-9341-89cc0af48c7f","Type":"ContainerStarted","Data":"626e97090ee253247a427f9275922b416f5ef1d35712f1216b66f67de16e551e"}
Oct 30 21:53:08.226050 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:08.226024    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:53:08.234616 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:08.234576    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podStartSLOduration=2.234563341 podStartE2EDuration="2.234563341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:08.231622561 +0000 UTC m=+806.889718784" watchObservedRunningTime="2026-10-30 21:53:08.234563341 +0000 UTC m=+806.892659563"
Oct 30 21:53:08.277458 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:08.274432    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:17.273624 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:17.273579    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:17.288288 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:17.288256    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:18.302108 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:18.302076    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:18.964500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:18.959857    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"]
Oct 30 21:53:19.048512 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.038059    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"]
Oct 30 21:53:19.048512 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.038211    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.112878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.112848    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-provider-credentials\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.113049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.112994    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-plugins-config-volume\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.113117 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.113069    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-routing-overlay-volume\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.113117 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.113103    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jnbc\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-kube-api-access-8jnbc\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.214211 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.214169    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-provider-credentials\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.214415 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.214309    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-plugins-config-volume\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.214415 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.214351    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-routing-overlay-volume\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.214415 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.214380    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jnbc\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-kube-api-access-8jnbc\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.237536 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.219087    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-routing-overlay-volume\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.237536 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.219799    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-plugins-config-volume\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.238298 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.238261    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-provider-credentials\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.244461 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.244427    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jnbc\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-kube-api-access-8jnbc\") pod \"payload-processing-external-model-6bd796bf-hfk8s\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") " pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.384967 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.383735    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:19.673749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:19.673725    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"]
Oct 30 21:53:19.675772 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:53:19.675739    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e48a25f_6bad_4142_891f_6789588c4029.slice/crio-c5aa5d374b409cae7df924d1562dea0d57a8659d6b18df22a9aece58f0724709 WatchSource:0}: Error finding container c5aa5d374b409cae7df924d1562dea0d57a8659d6b18df22a9aece58f0724709: Status 404 returned error can't find the container with id c5aa5d374b409cae7df924d1562dea0d57a8659d6b18df22a9aece58f0724709
Oct 30 21:53:20.300632 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:20.300600    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s" event={"ID":"8e48a25f-6bad-4142-891f-6789588c4029","Type":"ContainerStarted","Data":"6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43"}
Oct 30 21:53:20.300822 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:20.300659    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s" event={"ID":"8e48a25f-6bad-4142-891f-6789588c4029","Type":"ContainerStarted","Data":"c5aa5d374b409cae7df924d1562dea0d57a8659d6b18df22a9aece58f0724709"}
Oct 30 21:53:20.300822 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:20.300689    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:20.331378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:20.331322    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s" podStartSLOduration=2.331307086 podStartE2EDuration="2.331307086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:20.329554621 +0000 UTC m=+818.987650845" watchObservedRunningTime="2026-10-30 21:53:20.331307086 +0000 UTC m=+818.989403331"
Oct 30 21:53:21.874036 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.874011    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"]
Oct 30 21:53:21.878566 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.878540    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.901521 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.901005    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"]
Oct 30 21:53:21.951199 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951171    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.951347 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951205    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.951347 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951224    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.951424 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951349    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdtk2\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-kube-api-access-fdtk2\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.951458 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951437    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.951536 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951496    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.951571 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951537    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.951607 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951596    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:21.951651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:21.951640    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e91cee75-83ba-468a-a4c0-9c40820adedc-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.052794 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.052767    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.052956 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.052814    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e91cee75-83ba-468a-a4c0-9c40820adedc-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.052956 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.052871    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.052956 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.052902    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.052956 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.052926    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053146 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.052970    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdtk2\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-kube-api-access-fdtk2\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053146 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.053048    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053146 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.053079    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053371 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.053338    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.053370    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.053341    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.053424    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.053601    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.053688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.053682    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e91cee75-83ba-468a-a4c0-9c40820adedc-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.055282 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.055262    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.055793 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.055773    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.069129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.069107    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.070266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.070238    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdtk2\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-kube-api-access-fdtk2\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.194522 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.194493    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:22.487204 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.487038    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"]
Oct 30 21:53:22.490001 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:53:22.489977    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91cee75_83ba_468a_a4c0_9c40820adedc.slice/crio-a711ceb4365b6308406098cbd161cd6c4c3c262fa02ac6ae560610214d44b997 WatchSource:0}: Error finding container a711ceb4365b6308406098cbd161cd6c4c3c262fa02ac6ae560610214d44b997: Status 404 returned error can't find the container with id a711ceb4365b6308406098cbd161cd6c4c3c262fa02ac6ae560610214d44b997
Oct 30 21:53:22.492431 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.492399    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:22.492514 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.492459    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:22.492559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:22.492519    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:23.319156 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:23.319119    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" event={"ID":"e91cee75-83ba-468a-a4c0-9c40820adedc","Type":"ContainerStarted","Data":"d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4"}
Oct 30 21:53:23.319585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:23.319211    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" event={"ID":"e91cee75-83ba-468a-a4c0-9c40820adedc","Type":"ContainerStarted","Data":"a711ceb4365b6308406098cbd161cd6c4c3c262fa02ac6ae560610214d44b997"}
Oct 30 21:53:23.370493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:23.352707    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podStartSLOduration=2.352688644 podStartE2EDuration="2.352688644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:23.349653556 +0000 UTC m=+822.007749779" watchObservedRunningTime="2026-10-30 21:53:23.352688644 +0000 UTC m=+822.010784868"
Oct 30 21:53:24.196500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.195997    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:24.202492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.199581    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:53:24.202492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.199635    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:53:24.299633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.299601    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"]
Oct 30 21:53:24.303563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.303542    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.329015 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.328989    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"]
Oct 30 21:53:24.398075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398038    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.398273 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398091    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.398356 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398335    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrq2\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-kube-api-access-7wrq2\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.398460 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398431    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.398523 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398507    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.398569 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398557    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.398634 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398619    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.398821 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398806    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad0fdf8-6237-440f-b396-c1eebd36b982-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.398858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.398843    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.499662 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.499580    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.499662 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.499632    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad0fdf8-6237-440f-b396-c1eebd36b982-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.499662 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.499662    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.499961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.499687    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.499961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.499719    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.499961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.499849    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrq2\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-kube-api-access-7wrq2\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.499961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.499928    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.500164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.499975    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.500164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.500030    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.500164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.500122    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.500321 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.500226    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.500321 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.500278    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.500461 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.500449    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad0fdf8-6237-440f-b396-c1eebd36b982-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.500747 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.500621    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.502374 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.502350    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.502374 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.502367    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.513147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.513123    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.513517 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.513461    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrq2\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-kube-api-access-7wrq2\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:24.618216 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:24.618187    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:25.043590 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.043562    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"]
Oct 30 21:53:25.046439 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:53:25.046406    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad0fdf8_6237_440f_b396_c1eebd36b982.slice/crio-7be8bcdc362b1f32dbaee03f71902ebc12a247c1ca0583909c3c8dbd02e64b20 WatchSource:0}: Error finding container 7be8bcdc362b1f32dbaee03f71902ebc12a247c1ca0583909c3c8dbd02e64b20: Status 404 returned error can't find the container with id 7be8bcdc362b1f32dbaee03f71902ebc12a247c1ca0583909c3c8dbd02e64b20
Oct 30 21:53:25.048844 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.048812    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:25.048930 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.048888    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:25.048999 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.048934    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:25.079561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.079532    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:53:25.197583 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.196827    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:53:25.197583 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.196883    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:53:25.330983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.330880    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" event={"ID":"cad0fdf8-6237-440f-b396-c1eebd36b982","Type":"ContainerStarted","Data":"5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735"}
Oct 30 21:53:25.330983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.330949    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" event={"ID":"cad0fdf8-6237-440f-b396-c1eebd36b982","Type":"ContainerStarted","Data":"7be8bcdc362b1f32dbaee03f71902ebc12a247c1ca0583909c3c8dbd02e64b20"}
Oct 30 21:53:25.619363 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:25.619286    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:26.197918 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:26.197879    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:53:26.198166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:26.197937    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:53:26.620323 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:26.620227    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:53:26.620323 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:26.620302    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:53:27.198488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:27.198249    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:53:27.198488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:27.198324    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:53:27.621026 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:27.620931    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:53:27.621685 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:27.621652    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:53:28.199493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:28.198869    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:53:28.199493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:28.198954    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:53:28.619081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:28.618979    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:53:28.619081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:28.619051    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:53:29.195506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:29.195430    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:53:29.196023 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:29.195509    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 21:53:29.619448 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:29.619361    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:53:29.619448 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:29.619410    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:53:30.618752 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:30.618718    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:53:30.619165 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:30.618772    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:53:31.011604 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:31.011539    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podStartSLOduration=7.011520702 podStartE2EDuration="7.011520702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:25.380672545 +0000 UTC m=+824.038768769" watchObservedRunningTime="2026-10-30 21:53:31.011520702 +0000 UTC m=+829.669616926"
Oct 30 21:53:31.015912 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:31.015883    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"]
Oct 30 21:53:31.018980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:31.018930    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"]
Oct 30 21:53:31.025684 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:31.025662    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:31.195413 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:31.195376    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:53:31.195628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:31.195490    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:31.619261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:31.619225    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:53:31.619707 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:31.619292    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 21:53:32.220453 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:32.220420    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:53:32.220674 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:32.220500    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:32.221408 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:32.221389    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:32.221820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:32.221802    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:32.222540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:32.222524    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 21:53:33.660637 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:33.660597    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:53:33.661308 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:33.660661    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:34.146831 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:34.146798    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:34.171753 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:34.148568    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:34.175811 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:34.175777    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 21:53:34.648625 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:53:34.648586    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:53:36.017683 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.017630    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t" podUID="b864bc71-aa90-44b1-88ca-b5634874544d" containerName="payload-processing" containerID="cri-o://3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971" gracePeriod=25
Oct 30 21:53:36.020228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.020179    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s" podUID="8e48a25f-6bad-4142-891f-6789588c4029" containerName="payload-processing" containerID="cri-o://6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43" gracePeriod=25
Oct 30 21:53:36.275233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.275178    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:53:36.311952 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.311928    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:36.334884 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.334861    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-routing-overlay-volume\") pod \"8e48a25f-6bad-4142-891f-6789588c4029\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") "
Oct 30 21:53:36.335058 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335038    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-routing-overlay-volume\") pod \"b864bc71-aa90-44b1-88ca-b5634874544d\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") "
Oct 30 21:53:36.335160 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335078    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-plugins-config-volume\") pod \"b864bc71-aa90-44b1-88ca-b5634874544d\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") "
Oct 30 21:53:36.335160 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335106    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-kube-api-access-8jnbc\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-kube-api-access-8jnbc\") pod \"8e48a25f-6bad-4142-891f-6789588c4029\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") "
Oct 30 21:53:36.335281 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335238    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-routing-overlay-volume" pod "8e48a25f-6bad-4142-891f-6789588c4029" (UID: "8e48a25f-6bad-4142-891f-6789588c4029"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:36.335508 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335455    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-provider-credentials\") pod \"8e48a25f-6bad-4142-891f-6789588c4029\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") "
Oct 30 21:53:36.335608 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335546    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-kube-api-access-fgk7f\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-kube-api-access-fgk7f\") pod \"b864bc71-aa90-44b1-88ca-b5634874544d\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") "
Oct 30 21:53:36.335608 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335564    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-plugins-config-volume" pod "b864bc71-aa90-44b1-88ca-b5634874544d" (UID: "b864bc71-aa90-44b1-88ca-b5634874544d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:36.335608 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335570    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-provider-credentials\") pod \"b864bc71-aa90-44b1-88ca-b5634874544d\" (UID: \"b864bc71-aa90-44b1-88ca-b5634874544d\") "
Oct 30 21:53:36.335765 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.335638    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-plugins-config-volume\") pod \"8e48a25f-6bad-4142-891f-6789588c4029\" (UID: \"8e48a25f-6bad-4142-891f-6789588c4029\") "
Oct 30 21:53:36.336567 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.336245    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-plugins-config-volume" pod "8e48a25f-6bad-4142-891f-6789588c4029" (UID: "8e48a25f-6bad-4142-891f-6789588c4029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:36.336567 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.336508    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:36.336567 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.336532    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8e48a25f-6bad-4142-891f-6789588c4029-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:36.336567 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.336546    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:36.337147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.337116    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-routing-overlay-volume" pod "b864bc71-aa90-44b1-88ca-b5634874544d" (UID: "b864bc71-aa90-44b1-88ca-b5634874544d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:36.338315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.338228    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-kube-api-access-8jnbc" pod "8e48a25f-6bad-4142-891f-6789588c4029" (UID: "8e48a25f-6bad-4142-891f-6789588c4029"). InnerVolumeSpecName "kube-api-access-8jnbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:36.338315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.338249    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-provider-credentials" pod "8e48a25f-6bad-4142-891f-6789588c4029" (UID: "8e48a25f-6bad-4142-891f-6789588c4029"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:36.338315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.338280    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-provider-credentials" pod "b864bc71-aa90-44b1-88ca-b5634874544d" (UID: "b864bc71-aa90-44b1-88ca-b5634874544d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:36.339710 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.339678    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-kube-api-access-fgk7f" pod "b864bc71-aa90-44b1-88ca-b5634874544d" (UID: "b864bc71-aa90-44b1-88ca-b5634874544d"). InnerVolumeSpecName "kube-api-access-fgk7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:36.444503 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.438214    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b864bc71-aa90-44b1-88ca-b5634874544d-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:36.444503 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.438249    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jnbc\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-kube-api-access-8jnbc\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:36.444503 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.438263    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8e48a25f-6bad-4142-891f-6789588c4029-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:36.444503 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.438276    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgk7f\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-kube-api-access-fgk7f\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:36.444503 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.438289    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b864bc71-aa90-44b1-88ca-b5634874544d-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:36.638363 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.638275    4282 generic.go:356] "Generic (PLEG): container finished" podID="b864bc71-aa90-44b1-88ca-b5634874544d" containerID="3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971" exitCode=0
Oct 30 21:53:36.638363 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.638319    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"
Oct 30 21:53:36.638641 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.638357    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t" event={"ID":"b864bc71-aa90-44b1-88ca-b5634874544d","Type":"ContainerDied","Data":"3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971"}
Oct 30 21:53:36.638641 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.638422    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5c748545bd-msq2t" event={"ID":"b864bc71-aa90-44b1-88ca-b5634874544d","Type":"ContainerDied","Data":"09b9a2827fc542b2503c4657a10125d2acafd1414110dec9901cd419fd8a24d7"}
Oct 30 21:53:36.638641 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.638452    4282 scope.go:122] "RemoveContainer" containerID="3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971"
Oct 30 21:53:36.639861 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.639838    4282 generic.go:356] "Generic (PLEG): container finished" podID="8e48a25f-6bad-4142-891f-6789588c4029" containerID="6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43" exitCode=0
Oct 30 21:53:36.639975 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.639893    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"
Oct 30 21:53:36.639975 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.639916    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s" event={"ID":"8e48a25f-6bad-4142-891f-6789588c4029","Type":"ContainerDied","Data":"6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43"}
Oct 30 21:53:36.640080 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.639989    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s" event={"ID":"8e48a25f-6bad-4142-891f-6789588c4029","Type":"ContainerDied","Data":"c5aa5d374b409cae7df924d1562dea0d57a8659d6b18df22a9aece58f0724709"}
Oct 30 21:53:36.648180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.648053    4282 scope.go:122] "RemoveContainer" containerID="3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971"
Oct 30 21:53:36.648338 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:53:36.648320    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971\": container with ID starting with 3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971 not found: ID does not exist" containerID="3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971"
Oct 30 21:53:36.648426 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.648352    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971"} err="failed to get container status \"3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971\": rpc error: code = NotFound desc = could not find container \"3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971\": container with ID starting with 3b2f96a99e55f5bf29e1df82cc7092c40fe7f76ff1c52bd01831253977ba0971 not found: ID does not exist"
Oct 30 21:53:36.648426 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.648373    4282 scope.go:122] "RemoveContainer" containerID="6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43"
Oct 30 21:53:36.657563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.657530    4282 scope.go:122] "RemoveContainer" containerID="6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43"
Oct 30 21:53:36.657843 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:53:36.657818    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43\": container with ID starting with 6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43 not found: ID does not exist" containerID="6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43"
Oct 30 21:53:36.657934 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.657851    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43"} err="failed to get container status \"6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43\": rpc error: code = NotFound desc = could not find container \"6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43\": container with ID starting with 6c86b3f60090819815ff3dc263cf4faa182556298dbe0f2044e29045b7dc5f43 not found: ID does not exist"
Oct 30 21:53:36.665960 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.665938    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"]
Oct 30 21:53:36.674516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.674492    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6bd796bf-hfk8s"]
Oct 30 21:53:36.684375 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.684351    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"]
Oct 30 21:53:36.692409 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:36.692380    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5c748545bd-msq2t"]
Oct 30 21:53:37.463576 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:37.463541    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e48a25f-6bad-4142-891f-6789588c4029" path="/var/lib/kubelet/pods/8e48a25f-6bad-4142-891f-6789588c4029/volumes"
Oct 30 21:53:37.464021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:37.464005    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b864bc71-aa90-44b1-88ca-b5634874544d" path="/var/lib/kubelet/pods/b864bc71-aa90-44b1-88ca-b5634874544d/volumes"
Oct 30 21:53:40.663494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:40.662562    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:40.804612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:40.804561    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:41.227495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:41.224561    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:41.302671 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:41.301622    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:41.671143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:41.668600    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:41.671143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:41.669305    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:53:41.807506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:41.805143    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:41.807506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:41.805958    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:53:42.091065 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:42.091021    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:42.257495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:42.234919    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:42.283526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:42.259057    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:53:42.358493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:42.357489    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:42.360393 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:42.360369    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:42.687550 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:42.678750    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:42.869502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:42.867930    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:53:43.049589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:43.048692    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:53:43.125331 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:43.125292    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:53:43.126238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:43.126214    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:53:43.232551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:43.232519    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:53:43.283327 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:43.283294    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:53:43.365801 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:43.365713    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:53:43.380994 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:43.380967    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:53:43.432926 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:43.432281    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:53:49.108038 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.108000    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"]
Oct 30 21:53:49.108586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.108567    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b864bc71-aa90-44b1-88ca-b5634874544d" containerName="payload-processing"
Oct 30 21:53:49.108586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.108585    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b864bc71-aa90-44b1-88ca-b5634874544d" containerName="payload-processing"
Oct 30 21:53:49.108723 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.108601    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8e48a25f-6bad-4142-891f-6789588c4029" containerName="payload-processing"
Oct 30 21:53:49.108723 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.108606    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e48a25f-6bad-4142-891f-6789588c4029" containerName="payload-processing"
Oct 30 21:53:49.108723 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.108677    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b864bc71-aa90-44b1-88ca-b5634874544d" containerName="payload-processing"
Oct 30 21:53:49.108723 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.108686    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e48a25f-6bad-4142-891f-6789588c4029" containerName="payload-processing"
Oct 30 21:53:49.114121 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.114103    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.122197 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.122177    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"kube-root-ca.crt\""
Oct 30 21:53:49.122302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.122266    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"openshift-service-ca.crt\""
Oct 30 21:53:49.124482 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.124439    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"]
Oct 30 21:53:49.192201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.192181    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"default-dockercfg-brj97\""
Oct 30 21:53:49.192329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.192205    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b2e72\"/\"distinct-2-w2-2b2e72-kserve-self-signed-certs\""
Oct 30 21:53:49.198934 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.198902    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-dshm\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.199030 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.198941    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-model-cache\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.199075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.199061    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-home\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.199115 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.199086    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8756\" (UniqueName: \"kubernetes.io/projected/71a166e3-0bc4-4f50-af2a-a8b342fd9421-kube-api-access-n8756\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.199153 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.199118    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tls-certs\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.199259 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.199242    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tmp-dir\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.305418 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.304192    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tmp-dir\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.305418 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.304321    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-dshm\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.305418 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.304365    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-model-cache\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.305418 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.304525    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-home\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.305418 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.304557    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8756\" (UniqueName: \"kubernetes.io/projected/71a166e3-0bc4-4f50-af2a-a8b342fd9421-kube-api-access-n8756\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.305418 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.304603    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tls-certs\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.306336 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.306302    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-model-cache\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.309109 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.307043    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-home\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.309261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.308830    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tls-certs\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.309419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.309040    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tmp-dir\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.309997 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.309957    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-dshm\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.332501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.329250    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8756\" (UniqueName: \"kubernetes.io/projected/71a166e3-0bc4-4f50-af2a-a8b342fd9421-kube-api-access-n8756\") pod \"distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") " pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.453345 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.453313    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:49.735976 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.735949    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"]
Oct 30 21:53:49.737733 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:53:49.737698    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71a166e3_0bc4_4f50_af2a_a8b342fd9421.slice/crio-343bca23282b2ac78ccccfcc18aedda8d5e103347830881a15380ef21615fb0b WatchSource:0}: Error finding container 343bca23282b2ac78ccccfcc18aedda8d5e103347830881a15380ef21615fb0b: Status 404 returned error can't find the container with id 343bca23282b2ac78ccccfcc18aedda8d5e103347830881a15380ef21615fb0b
Oct 30 21:53:49.948584 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.948532    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:53:49.985302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.985213    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" event={"ID":"71a166e3-0bc4-4f50-af2a-a8b342fd9421","Type":"ContainerStarted","Data":"7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331"}
Oct 30 21:53:49.985302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:49.985265    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" event={"ID":"71a166e3-0bc4-4f50-af2a-a8b342fd9421","Type":"ContainerStarted","Data":"343bca23282b2ac78ccccfcc18aedda8d5e103347830881a15380ef21615fb0b"}
Oct 30 21:53:50.025129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:50.025066    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podStartSLOduration=1.025046872 podStartE2EDuration="1.025046872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:50.023087985 +0000 UTC m=+848.681184208" watchObservedRunningTime="2026-10-30 21:53:50.025046872 +0000 UTC m=+848.683143097"
Oct 30 21:53:50.453735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:50.453671    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:51.957456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:51.957423    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"]
Oct 30 21:53:51.969510 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:51.969327    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:51.988084 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:51.988061    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"]
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134526    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134588    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfdfc\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-kube-api-access-pfdfc\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134639    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134693    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134744    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134801    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134834    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134941    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.153500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.134997    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278051    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278119    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278161    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278206    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfdfc\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-kube-api-access-pfdfc\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278258    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278303    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278348    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278661 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278406    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.278661 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.278441    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.294045 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.294005    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.319230 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.319143    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.319890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.319851    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.322000 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.321980    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.336287 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.336259    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.391513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.391124    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.391513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.391385    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfdfc\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-kube-api-access-pfdfc\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.393268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.393209    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.395803 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.395604    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.679010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.678982    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:52.973510 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.973482    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"]
Oct 30 21:53:52.975573 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:53:52.975544    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f3ad33_a4b4_4dea_aa9f_e4773f7675cc.slice/crio-c768a3e690cb8fcc13c2882f0e335557fb309b5fe3825c043f00674faccd3566 WatchSource:0}: Error finding container c768a3e690cb8fcc13c2882f0e335557fb309b5fe3825c043f00674faccd3566: Status 404 returned error can't find the container with id c768a3e690cb8fcc13c2882f0e335557fb309b5fe3825c043f00674faccd3566
Oct 30 21:53:52.980093 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.980052    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:52.980204 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.980133    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:52.980204 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:52.980165    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:53:53.119250 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:53.119220    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" event={"ID":"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc","Type":"ContainerStarted","Data":"bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998"}
Oct 30 21:53:53.119414 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:53.119267    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" event={"ID":"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc","Type":"ContainerStarted","Data":"c768a3e690cb8fcc13c2882f0e335557fb309b5fe3825c043f00674faccd3566"}
Oct 30 21:53:53.148379 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:53.148319    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podStartSLOduration=2.148302269 podStartE2EDuration="2.148302269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:53.144927702 +0000 UTC m=+851.803023925" watchObservedRunningTime="2026-10-30 21:53:53.148302269 +0000 UTC m=+851.806398493"
Oct 30 21:53:53.682829 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:53.682786    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:53:54.681966 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:54.681931    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:53:54.682442 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:54.681987    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:53:55.680554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:55.680516    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:53:55.680774 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:55.680607    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:53:56.680330 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:56.679925    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:53:56.680330 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:56.679998    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:53:57.680542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:57.679848    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:53:57.680542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:57.679920    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:53:58.682496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:58.680850    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:53:58.682496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:58.680910    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:53:59.453978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:59.453945    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:59.466960 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:59.466933    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:53:59.679446 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:59.679407    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:53:59.679640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:53:59.679457    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 21:54:00.213871 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:00.213841    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:00.684212 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:00.684183    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:54:00.684738 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:00.684719    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:54:00.685493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:00.685456    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 21:54:04.648177 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:54:04.648142    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:54:07.266215 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.266183    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"]
Oct 30 21:54:07.295493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.280251    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.413214 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.413175    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.413543 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.413520    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2sg\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-kube-api-access-nl2sg\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.413969 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.413947    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.414133 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.414115    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.420498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.414389    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.420666 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.420645    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.420799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.420783    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.434493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.420899    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.435024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.434999    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.435308 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.435289    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"]
Oct 30 21:54:07.537535 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.486707    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"]
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542573    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542621    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542663    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542697    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542736    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542789    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542817    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2sg\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-kube-api-access-nl2sg\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542867    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.542909    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.566495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.543756    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.617495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.575614    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.617495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.582987    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.617495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.583259    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.617495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.609131    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.617495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.609182    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.700510 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.686104    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.700878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.700852    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.729498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.729440    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.741561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.741532    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.780767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.773553    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1660a82-f833-45b3-883c-9a34c1ec42a6-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.780767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.773673    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.780767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.773755    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.780767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.773890    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.780767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.773992    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh574\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-kube-api-access-rh574\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.780767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.731343    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"]
Oct 30 21:54:07.823585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.823511    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"]
Oct 30 21:54:07.842494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.838823    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"]
Oct 30 21:54:07.870494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.826836    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:07.878494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.824994    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.889594 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.853750    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2sg\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-kube-api-access-nl2sg\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:07.906598 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.880582    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:07.930629 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.930591    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.930930 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.930900    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh574\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-kube-api-access-rh574\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.931061 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.931045    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.937523 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.931160    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:07.942570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.942539    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:07.968530 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.946535    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:07.992497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.968752    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.992902 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.992874    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:07.993092 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.993070    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.015195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.014956    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.015457 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.015418    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1660a82-f833-45b3-883c-9a34c1ec42a6-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.044886 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.044853    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:08.072580 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.974112    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.074354 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.047014    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:08.078541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.047060    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.078736 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.078716    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9sh\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-kube-api-access-fm9sh\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.083543 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.078883    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.174100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.174045    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.174640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.174598    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.176179 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.176057    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.176883 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.968830    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.177057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.048878    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1660a82-f833-45b3-883c-9a34c1ec42a6-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.177181 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:07.996367    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh574\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-kube-api-access-rh574\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.177284 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.047766    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.177384 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.001012    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.179489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.175901    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.179712 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.047276    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.198517 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.189082    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.210498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.199085    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:08.240499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.239611    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:08.240818 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.240785    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.299931    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.300025    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.300111    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.300137    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.300166    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.300275    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.300310    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.300348    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9sh\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-kube-api-access-fm9sh\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.300682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.300395    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.325430 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.325363    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.347973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.325948    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.348185 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.331709    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.348307 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.347509    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.348406 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.347903    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.365705 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.356637    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.366072 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.366047    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.366401 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.366382    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:08.377491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.367758    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:08.382492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.379264    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:08.382492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.379355    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:54:08.390502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.385972    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.390502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.389964    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:08.461097 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.461067    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9sh\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-kube-api-access-fm9sh\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.514495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.511541    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:08.688978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.688226    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:08.864987 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.864630    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:08.864987 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:08.864693    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:09.247545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.241417    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:09.247545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.242165    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:09.247545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.243875    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:09.247545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.244263    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:09.257077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.256936    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:09.257077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.257003    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:09.324318 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.323327    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:09.324318 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.323385    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:09.380614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.380577    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:09.450585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.433394    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:09.520312 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:54:09.520244    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a17208_d1ac_4c41_b5f4_8a4e3bbd3a97.slice/crio-ec451ae249e328b29a37c5cc29651b500aec286ee7ef45d32b7cf58a5ecac385 WatchSource:0}: Error finding container ec451ae249e328b29a37c5cc29651b500aec286ee7ef45d32b7cf58a5ecac385: Status 404 returned error can't find the container with id ec451ae249e328b29a37c5cc29651b500aec286ee7ef45d32b7cf58a5ecac385
Oct 30 21:54:09.526925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.522644    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.526925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.522727    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.526925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.522796    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.629162    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.629218    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.629249    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.629281    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"]
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.629322    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"]
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.629350    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"]
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.636602    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.636684    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.638261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.636726    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.643345 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.643319    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"]
Oct 30 21:54:09.643501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.643368    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"]
Oct 30 21:54:09.645492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.644821    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.653322 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.652686    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" event={"ID":"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97","Type":"ContainerStarted","Data":"ec451ae249e328b29a37c5cc29651b500aec286ee7ef45d32b7cf58a5ecac385"}
Oct 30 21:54:09.653322 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.652732    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"]
Oct 30 21:54:09.653322 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.652862    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.795132 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.795103    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"]
Oct 30 21:54:09.800708 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:54:09.800544    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d6edcc_0583_4a8b_ac74_9e4bcc4501a3.slice/crio-c94f071f097e22a33a0ef347701d9ff271f9bef31f708e7d90dacdeb3c4af432 WatchSource:0}: Error finding container c94f071f097e22a33a0ef347701d9ff271f9bef31f708e7d90dacdeb3c4af432: Status 404 returned error can't find the container with id c94f071f097e22a33a0ef347701d9ff271f9bef31f708e7d90dacdeb3c4af432
Oct 30 21:54:09.803831 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.803801    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.803932 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.803863    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.803932 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.803893    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:09.816741 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.816714    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.816824 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.816770    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.816824 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.816814    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.817037 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.816966    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.817037 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817002    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.817207 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817081    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.817207 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817117    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.817207 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817184    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.817364 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817215    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6e9352-4eac-4838-9640-612076e9b0e6-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.817364 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817255    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb9mr\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-kube-api-access-kb9mr\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.817364 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817307    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.817364 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817334    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.817991 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817889    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.817991 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817933    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkzf\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-kube-api-access-4pkzf\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.817991 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.817962    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.818180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.818016    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.818180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.818045    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.818180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.818085    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.921834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.921784    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.922132 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.922110    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.922295 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.922276    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.922448 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.922430    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.922638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.922617    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.922853 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.922833    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.922988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.922972    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.923139 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.923121    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.923270 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.923254    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6e9352-4eac-4838-9640-612076e9b0e6-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.923411 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.923391    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb9mr\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-kube-api-access-kb9mr\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.923579 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.923560    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.923706 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.923691    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.923877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.923858    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.924005 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.923988    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkzf\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-kube-api-access-4pkzf\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.924139 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.924106    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.924280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.924261    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.924408 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.924393    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.924557 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.924540    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.925191 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.925168    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.926294 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.926245    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.926943 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.926554    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.939163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.928395    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.939163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.928512    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.939163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.928730    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.939163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.928775    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.939163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.929100    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.939163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.929207    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.939163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.929431    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6e9352-4eac-4838-9640-612076e9b0e6-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.942815 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.942301    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.944528 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.944140    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.945557 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.945496    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.945557 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.945516    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.947060 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.947033    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:09.951875 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.951239    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.951875 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.951815    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb9mr\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-kube-api-access-kb9mr\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:09.953069 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:09.953028    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkzf\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-kube-api-access-4pkzf\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:10.022352 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.022268    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:10.097506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.097430    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:10.416449 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.416250    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"]
Oct 30 21:54:10.434980 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:54:10.434939    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4918fb2b_e3d4_4584_93f0_46cbdad0e656.slice/crio-e976cbf5eeb847eed99838d01afbbf544bd01fd0bb39e3fe748a5c2e8df5e0fd WatchSource:0}: Error finding container e976cbf5eeb847eed99838d01afbbf544bd01fd0bb39e3fe748a5c2e8df5e0fd: Status 404 returned error can't find the container with id e976cbf5eeb847eed99838d01afbbf544bd01fd0bb39e3fe748a5c2e8df5e0fd
Oct 30 21:54:10.440731 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.437981    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:10.440731 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.438051    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:10.440731 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.438083    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:10.523968 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.523658    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"]
Oct 30 21:54:10.528590 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.528516    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:10.529825 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.528826    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:10.529825 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.528889    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:10.931962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.912957    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" event={"ID":"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3","Type":"ContainerStarted","Data":"699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425"}
Oct 30 21:54:10.931962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:10.913085    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" event={"ID":"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3","Type":"ContainerStarted","Data":"c94f071f097e22a33a0ef347701d9ff271f9bef31f708e7d90dacdeb3c4af432"}
Oct 30 21:54:11.024501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.020598    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podStartSLOduration=4.020571083 podStartE2EDuration="4.020571083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:11.002181772 +0000 UTC m=+869.660277996" watchObservedRunningTime="2026-10-30 21:54:11.020571083 +0000 UTC m=+869.678667298"
Oct 30 21:54:11.032065 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.032022    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" event={"ID":"8e6e9352-4eac-4838-9640-612076e9b0e6","Type":"ContainerStarted","Data":"d5c54a4179c19ef9095463e85411772175d4cd154356ab9b2c487e2ec0e1908b"}
Oct 30 21:54:11.151314 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.151217    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" event={"ID":"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97","Type":"ContainerStarted","Data":"17866631305f375d2d8466be634ad51e6d3e33bae02944873bbf0fdbdbc27062"}
Oct 30 21:54:11.199822 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.199737    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podStartSLOduration=4.199717147 podStartE2EDuration="4.199717147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:11.196941603 +0000 UTC m=+869.855037827" watchObservedRunningTime="2026-10-30 21:54:11.199717147 +0000 UTC m=+869.857813371"
Oct 30 21:54:11.243573 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.243535    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" event={"ID":"4918fb2b-e3d4-4584-93f0-46cbdad0e656","Type":"ContainerStarted","Data":"66794b45ffe05dbd78307ae0412993aa70de6d6b435426c439ca788ef3ac090a"}
Oct 30 21:54:11.243809 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.243793    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" event={"ID":"4918fb2b-e3d4-4584-93f0-46cbdad0e656","Type":"ContainerStarted","Data":"e976cbf5eeb847eed99838d01afbbf544bd01fd0bb39e3fe748a5c2e8df5e0fd"}
Oct 30 21:54:11.263905 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.263772    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" event={"ID":"a1660a82-f833-45b3-883c-9a34c1ec42a6","Type":"ContainerStarted","Data":"c6813dd14b18bc5a3e3b18891299044a9be596991648f76d324c7dc39ec85a74"}
Oct 30 21:54:11.308538 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.287529    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" event={"ID":"a1660a82-f833-45b3-883c-9a34c1ec42a6","Type":"ContainerStarted","Data":"cf7a2bfbe3797df65f37d8f4fd276971c8e233c807bfd7ea5bbbd90b639da433"}
Oct 30 21:54:11.308843 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.287401    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podStartSLOduration=2.287379392 podStartE2EDuration="2.287379392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:11.285697585 +0000 UTC m=+869.943793809" watchObservedRunningTime="2026-10-30 21:54:11.287379392 +0000 UTC m=+869.945475615"
Oct 30 21:54:11.381504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.380635    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:11.381703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.381551    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:11.381703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.381600    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:11.629258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.629223    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:11.644264 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.644155    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:11.644513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:11.644451    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:12.030495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.028356    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:12.059513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.053602    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:12.059513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.053661    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:12.201669 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.201221    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:12.201669 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.201343    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:54:12.201669 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.201384    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:54:12.380559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.367513    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:12.380559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.367573    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:12.398496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.390685    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" event={"ID":"8e6e9352-4eac-4838-9640-612076e9b0e6","Type":"ContainerStarted","Data":"b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf"}
Oct 30 21:54:12.482522 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.474917    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podStartSLOduration=5.474894212 podStartE2EDuration="5.474894212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:11.364898496 +0000 UTC m=+870.022994771" watchObservedRunningTime="2026-10-30 21:54:12.474894212 +0000 UTC m=+871.132990436"
Oct 30 21:54:12.536502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.520920    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:12.536502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.520980    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:12.556993 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.556599    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:54:12.625620 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.625571    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:12.694358 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.692386    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:54:12.694901 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:12.691333    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:13.031568 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.031337    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:13.031568 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.031398    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:13.060501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.056699    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:13.107799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.107329    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:13.107799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.107485    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:13.107799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.107532    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:13.200935 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.200897    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:54:13.201250 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.201213    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:54:13.618492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.560502    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:54:13.688511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.594569    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:13.688511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.596048    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:13.688511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.685457    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:13.688511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.596307    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:13.688511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.685771    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:13.688511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.686848    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:13.797643 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.797589    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:13.846545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.823049    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:13.925553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:13.893545    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:54:14.041763 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.041727    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:14.085878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.085209    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:14.085878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.043785    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:14.085878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.085375    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:14.085878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.085420    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:14.085878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.085454    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:54:14.101201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.101093    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:14.101201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.101147    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:14.121841 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.110325    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:14.214160 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.182335    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podStartSLOduration=5.182314021 podStartE2EDuration="5.182314021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:12.453865687 +0000 UTC m=+871.111961910" watchObservedRunningTime="2026-10-30 21:54:14.182314021 +0000 UTC m=+872.840410247"
Oct 30 21:54:14.263921 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.233006    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:54:14.264203 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.264171    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:54:14.264369 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.233505    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:54:14.530227 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.530133    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:14.534488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.530561    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:14.558492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.535242    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:14.588610 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.588562    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:14.687314 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.687262    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:14.810538 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.788533    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:14.838204 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.837444    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:14.868502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.799176    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:54:14.869405 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.798369    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:14.950525 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.873066    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:54:15.087555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:14.893807    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:15.087555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.068614    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:15.087555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.084300    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:15.464074 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.448614    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:15.464074 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.448680    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:15.481724 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.481681    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:15.537370 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.537332    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:54:15.537672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.537644    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:54:15.563941 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.563905    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:15.565579 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.565546    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:15.587488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.584487    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:15.587488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.584538    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:15.648498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.642283    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:54:15.849359 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.848557    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:15.873598 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.873554    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:15.915772 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:15.915733    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:16.025085 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.025038    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:16.025362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.025332    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:16.094656 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.094607    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:16.102049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.101953    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:16.102323 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.102283    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:16.203027 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.202982    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:54:16.203304 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.203254    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:54:16.369979 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.369888    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:16.370254 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.370224    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:16.520500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.519865    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:16.520500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.519925    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:16.534614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.534573    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:16.595640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.595588    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:16.852633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.852577    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:16.880613 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.880566    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:16.946489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:16.936614    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:17.039491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.023340    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:17.039491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.023417    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:17.099502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.099363    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:17.103544 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.100317    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:17.103544 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.100427    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:17.208612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.208574    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:54:17.208890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.208862    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:54:17.368932 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.368840    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:17.369190 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.369159    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:17.520899 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.520859    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:17.521173 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.521145    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:17.541848 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.541806    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:17.885501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.884091    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:17.958494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.950563    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:54:17.958494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:17.951831    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:18.026495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.024029    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:18.026495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.024106    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:18.103495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.103067    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:18.103495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.102634    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:18.103495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.103215    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:18.203495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.200815    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:54:18.203495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.200872    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:54:18.276500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.275815    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Oct 30 21:54:18.355595 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.354736    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:18.380499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.373970    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:18.380499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.374046    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:18.410149 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.410042    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:18.534573 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.528525    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:18.534573 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.528593    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:18.553500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.549231    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:18.681313 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.681208    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:54:18.957674 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.950763    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:18.993518 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:18.973544    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:19.065704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.061883    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:19.065704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.061942    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:19.104539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.102871    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:19.104539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.102926    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:19.108320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.108254    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:19.201899 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.201862    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 21:54:19.202146 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.202120    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 21:54:19.355352 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.355267    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:19.371797 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.371769    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 30 21:54:19.372046 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.372019    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 30 21:54:19.711510 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.687705    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:19.711510 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.689593    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:19.741504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.726232    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:54:19.773418 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.755453    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:19.773709 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.773683    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:19.837819 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.837785    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:54:19.934642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.934611    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:54:19.957177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:19.954551    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:20.057834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.057642    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:20.057834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.057703    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:20.173494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.163427    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 30 21:54:20.235495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.235230    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:20.235495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.235311    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:20.509505 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.505844    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 30 21:54:20.551522 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.551449    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 21:54:20.551793 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.551751    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 21:54:20.697139 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.695708    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:20.715760 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.715733    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:54:20.833638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.820610    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:20.833638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.822899    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:20.833638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.831550    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:54:20.833638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.832427    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:20.838238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:20.838049    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 21:54:21.029887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:21.029838    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 21:54:21.030211 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:21.030166    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 21:54:21.150033 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:21.149949    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:21.226905 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:21.221374    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:21.226905 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:21.225146    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 21:54:21.228981 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:21.228956    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:21.229300 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:21.229265    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:22.100500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.099624    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 21:54:22.100500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.099683    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 21:54:22.518492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.517803    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"]
Oct 30 21:54:22.536788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.536748    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:22.537064 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.537035    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:22.557497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.546072    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.559042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.559013    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"]
Oct 30 21:54:22.623642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623279    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.623642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623343    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.623642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623447    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgknc\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-kube-api-access-cgknc\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.623642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623513    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.623642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623581    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.623642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623622    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.624131 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623651    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.624131 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623714    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.624131 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.623742    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.695489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.695279    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830129    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgknc\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-kube-api-access-cgknc\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830212    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830278    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830324    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830361    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830424    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830456    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830574    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.830621    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.904537    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.905932    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.907280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.906413    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:22.937410 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:22.937374    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:23.006484 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.002504    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:23.025450 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.016721    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:23.070786 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.070739    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:54:23.089504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.075806    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:23.089504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.075879    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:23.097502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.096809    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:23.097502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.097296    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:23.118516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.101062    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:23.219238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.219204    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:23.303490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.227823    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgknc\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-kube-api-access-cgknc\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:23.303490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.256358    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"]
Oct 30 21:54:23.503550 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.486265    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:23.546074 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.546043    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:23.556330 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.556296    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.630303 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.601667    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:23.630614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.630576    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:23.657434 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.657396    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.670363 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.670331    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.670630 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.670607    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.675834 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.675807    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75810534-d64f-4003-89e9-1222cb927848-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.676075 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.676055    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75810534-d64f-4003-89e9-1222cb927848-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.676912 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.676890    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.685016 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.684992    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.685317 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.685292    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xwm\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-kube-api-access-n2xwm\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.685542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.685523    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.693527 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.666597    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:23.735197 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.702791    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:23.857934 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.857888    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:23.886660 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.886631    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:23.888729 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.888706    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"]
Oct 30 21:54:23.888937 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.888924    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:23.889109 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.889096    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 21:54:23.889260 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.889247    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.898684    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xwm\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-kube-api-access-n2xwm\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.898783    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.898851    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.898902    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.898969    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.898996    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75810534-d64f-4003-89e9-1222cb927848-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.899046    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75810534-d64f-4003-89e9-1222cb927848-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.899172    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.899206    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.901032    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.911492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.901286    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.968813 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.968777    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.982915 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.981210    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:23.983313 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:23.982824    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75810534-d64f-4003-89e9-1222cb927848-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:24.010873 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.010819    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:24.011180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.011134    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:24.012207 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.012173    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:24.083074 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.082984    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:24.084242 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.083593    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:24.106609 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.106570    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:24.117282 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.117236    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:24.118124 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.118098    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:24.136108 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.118288    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:24.154557 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.123591    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:24.154871 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.154848    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:24.155855 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.123644    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:24.171391 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.171369    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:54:24.341675 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.341588    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:24.341936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.341903    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:24.502533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.501966    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75810534-d64f-4003-89e9-1222cb927848-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:24.608958 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.599122    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:24.720501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.691317    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 21:54:24.720501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.719515    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xwm\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-kube-api-access-n2xwm\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:24.796186 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.782581    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:24.806720 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:24.773721    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:25.035887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.028263    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:25.035887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.028324    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:25.035887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.033867    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:25.035887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.034443    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:25.187555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.169534    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:25.206700 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.204722    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:25.206700 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.205545    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:25.234352 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.215390    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:54:25.252495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.248598    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:25.420609 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.416681    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 21:54:25.424501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.424441    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:25.591668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.567860    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:25.591668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.567960    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:25.591668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.568038    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:25.591668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.582636    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:54:25.591668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.582683    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:25.591668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.582720    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:54:25.627734 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.624052    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"]
Oct 30 21:54:25.725845 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.725811    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:25.733553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.733529    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:25.747489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.737109    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 21:54:25.836104 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.836060    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:25.836194 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.836159    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:25.836242 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.836204    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:25.845191 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:25.844887    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"]
Oct 30 21:54:26.183364 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.183315    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" event={"ID":"75810534-d64f-4003-89e9-1222cb927848","Type":"ContainerStarted","Data":"ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550"}
Oct 30 21:54:26.183587 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.183378    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" event={"ID":"75810534-d64f-4003-89e9-1222cb927848","Type":"ContainerStarted","Data":"81cf9184f5f0a6da53e4d4524014c3237f6b6d6de3375ce05a0e005ff4d5a492"}
Oct 30 21:54:26.185399 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.185356    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" event={"ID":"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d","Type":"ContainerStarted","Data":"51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456"}
Oct 30 21:54:26.185568 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.185406    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" event={"ID":"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d","Type":"ContainerStarted","Data":"6f79ca46d239ed4fc9ba436eee07986566aa7019b8dd83d30bc78128628e2f59"}
Oct 30 21:54:26.217176 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.217113    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podStartSLOduration=4.217092969 podStartE2EDuration="4.217092969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:26.214909527 +0000 UTC m=+884.873005752" watchObservedRunningTime="2026-10-30 21:54:26.217092969 +0000 UTC m=+884.875189193"
Oct 30 21:54:26.258850 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.258795    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podStartSLOduration=4.258777555 podStartE2EDuration="4.258777555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:26.257616617 +0000 UTC m=+884.915712838" watchObservedRunningTime="2026-10-30 21:54:26.258777555 +0000 UTC m=+884.916873780"
Oct 30 21:54:26.270412 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.270382    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:26.271857 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.271543    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:54:26.272065 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.272039    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:54:26.667332 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.667304    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:26.668525 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.668503    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:54:26.668623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:26.668550    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:54:27.283494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:27.282671    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:54:27.283494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:27.282730    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:54:27.700500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:27.692302    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:54:27.700500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:27.692358    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:54:28.275936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:28.275896    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:54:28.276231 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:28.276197    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:54:28.674553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:28.673850    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:54:28.674553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:28.673906    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:54:29.273493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:29.272852    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:54:29.273493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:29.272913    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:54:29.675546 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:29.670971    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:54:29.675546 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:29.671044    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:54:30.275908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:30.275872    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:54:30.276668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:30.276638    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:54:30.674550 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:30.667774    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:54:30.674550 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:30.667836    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:54:31.283496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:31.273877    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:54:31.283496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:31.273939    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:54:31.672558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:31.669240    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:54:31.672558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:31.669303    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:54:32.290516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:32.285897    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:54:32.290516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:32.285959    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:54:32.668012 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:32.667979    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:54:32.668179 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:32.668047    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 21:54:33.270459 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:33.270421    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:54:33.270693 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:33.270511    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 21:54:33.894878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:33.894842    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:33.895344 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:33.895297    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:33.896048 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:33.896025    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 21:54:34.274614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:34.274587    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:34.275061 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:34.275042    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:34.275838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:34.275821    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 21:54:34.649377 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:54:34.649297    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:54:37.881115 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.881083    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"]
Oct 30 21:54:37.885408 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.885388    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.905524 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905498    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.905633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905561    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.905705 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905690    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.905810 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905789    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.905857 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905841    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.905926 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905914    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.905968 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905938    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.905968 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905956    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.906053 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.905990    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzhhq\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-kube-api-access-xzhhq\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:37.961399 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:37.961370    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"]
Oct 30 21:54:38.007147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007117    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007204    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007266    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007446 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007349    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007446 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007383    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007446 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007408    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007634 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007461    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzhhq\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-kube-api-access-xzhhq\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007634 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007512    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007634 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007563    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007802 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007765    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.007933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007903    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.008009 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.007981    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.008182 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.008156    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.008361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.008314    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.009829 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.009809    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.010065 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.010045    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.025374 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.025345    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.026839 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.026813    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzhhq\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-kube-api-access-xzhhq\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.199514 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.199461    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:38.487544 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.487517    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"]
Oct 30 21:54:38.488726 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:54:38.488690    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode348f2cc_4f24_47ad_b0c6_41d667b29f78.slice/crio-dc272a7ff58a6c03d0b86cc6a204f96eeab78bfd076d86fab423ff6884e2dd00 WatchSource:0}: Error finding container dc272a7ff58a6c03d0b86cc6a204f96eeab78bfd076d86fab423ff6884e2dd00: Status 404 returned error can't find the container with id dc272a7ff58a6c03d0b86cc6a204f96eeab78bfd076d86fab423ff6884e2dd00
Oct 30 21:54:38.491206 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.491171    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:38.491259 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.491238    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:38.491302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:38.491276    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:39.662494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:39.659724    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" event={"ID":"e348f2cc-4f24-47ad-b0c6-41d667b29f78","Type":"ContainerStarted","Data":"3768f4a2356dd3a2b4ffa71a332efc5e934639992e590411fe464026ddcf16a4"}
Oct 30 21:54:39.662494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:39.659797    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" event={"ID":"e348f2cc-4f24-47ad-b0c6-41d667b29f78","Type":"ContainerStarted","Data":"dc272a7ff58a6c03d0b86cc6a204f96eeab78bfd076d86fab423ff6884e2dd00"}
Oct 30 21:54:39.677289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:39.677219    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podStartSLOduration=2.677194191 podStartE2EDuration="2.677194191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:39.663389581 +0000 UTC m=+898.321485804" watchObservedRunningTime="2026-10-30 21:54:39.677194191 +0000 UTC m=+898.335290417"
Oct 30 21:54:40.120625 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.120569    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:40.203749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.203713    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:40.204174 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.204148    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:40.204409 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.204365    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:40.454440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.454397    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:40.539495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.538567    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:40.539495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.538905    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:40.577688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.576581    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:40.733412 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.733305    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:40.951497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:40.946587    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:41.085563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:41.081977    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:41.085563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:41.082046    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:41.214494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:41.206554    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:41.214494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:41.206615    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:41.448749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:41.448645    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:41.653980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:41.610748    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:54:41.653980 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:41.642514    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:41.992766 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:41.983615    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:42.143480 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:42.143420    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:54:42.572800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:42.571670    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:42.592548 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:42.590834    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:42.592548 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:42.590927    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:42.616245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:42.616204    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:42.644878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:42.644837    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:42.992251 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:42.992194    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:43.217640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.217607    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:43.229491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.219548    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:43.342416 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:54:43.342322    4282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.717s"
Oct 30 21:54:43.342744 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.342726    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:54:43.342921 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.342907    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:54:43.343095 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.343081    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:43.343257 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.343244    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:54:43.548351 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.548306    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:43.554044 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.554008    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:43.619660 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.619559    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:54:43.842454 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.842406    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:43.889562 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.889405    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:44.016132 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:43.895578    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:44.033576 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.033549    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:44.098950 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.098908    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:44.302497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.288306    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:44.303139 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.303100    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:44.499492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.499347    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:54:44.506429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.506402    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:44.508822 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.508798    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:54:44.556890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.556799    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:44.564861 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.564831    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:44.650329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.650299    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:44.934987 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.933332    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:44.938281 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:44.937757    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:54:45.200781 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:45.200691    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:45.200781 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:45.200764    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:46.199922 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:46.199891    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:46.200343 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:46.199943    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:47.200498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:47.200426    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:47.200938 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:47.200515    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:48.200492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:48.200440    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:48.200674 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:48.200513    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:49.139031 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:49.138991    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:54:49.200487 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:49.200440    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 21:54:49.200681 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:49.200523    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 21:54:50.303493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:50.293162    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:50.303493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:50.294598    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:50.303493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:50.299686    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 21:54:51.404790 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.370787    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"]
Oct 30 21:54:51.756042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.755996    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-podinfo\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.810494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.793716    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-credential-socket\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.810877 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.810846    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-socket\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.868500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.843643    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-envoy\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.868838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.868803    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-token\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.869055 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.869033    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47z9m\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-kube-api-access-47z9m\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.869230 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.869212    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-certs\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.882598 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.882562    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istiod-ca-cert\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.890597 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.890562    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-data\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.911494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.849115    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:51.933495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.928079    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-689a2563-openshift-default-dockercfg-cbvr7\""
Oct 30 21:54:51.950493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.939548    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:51.950493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.941298    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"]
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999317    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-podinfo\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999387    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-credential-socket\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999454    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-socket\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999579    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-envoy\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999608    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-token\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999671    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47z9m\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-kube-api-access-47z9m\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999727    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-certs\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999756    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istiod-ca-cert\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:51.999797    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-data\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.001493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.000407    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-data\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.059628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.026681    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-certs\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.059628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.026970    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-socket\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.059628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.027246    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-credential-socket\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.059628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.027491    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istiod-ca-cert\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.070487 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.067271    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-envoy\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.095563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.083047    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-podinfo\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.103684 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.103634    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:52.117102 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.117070    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47z9m\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-kube-api-access-47z9m\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.117804 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.117776    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-token\") pod \"e2e-mt-689a2563-openshift-default-d86655557-2ktpm\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") " pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.181495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.166938    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:52.275010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.274659    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:52.275010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.274718    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:52.870847 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.870795    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:52.931626 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.931573    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:54:52.932297 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.932265    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:54:52.954847 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.954815    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:53.009513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.984577    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:54:53.165371 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:52.954752    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:53.165371 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:53.154725    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:54:53.172504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:53.168488    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:53.899495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:53.884979    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:53.931663 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:53.904308    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:53.931663 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:53.904386    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:53.931663 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:53.914933    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:54.032810 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.032764    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:54.117719 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.070000    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:54:54.117719 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.091904    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:54:54.160720 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.160619    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:54.194487 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.181225    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:54.194487 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.181396    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:54.194487 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.181518    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:54.194487 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.191367    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:54.232495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.220131    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:54.232495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.220229    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:54:54.232495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.220287    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:54:54.232495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.220342    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:54:54.444598 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.287374    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"]
Oct 30 21:54:54.563715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.557586    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:54.563715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.557674    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:54.833879 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.833780    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 21:54:54.852514 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.848145    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc" podUID="76d92e07-0511-4c67-abdc-bdc12605768b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 30 21:54:54.880375 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.879590    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:54.881138 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.881103    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:54.927817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.927763    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:54.937660 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.937622    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:54.937913 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:54.937885    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:55.102245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:55.080844    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:55.102245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:55.081220    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:55.102245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:55.100420    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:55.172703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:55.172662    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 21:54:55.217529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:55.196835    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:55.940694 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:55.940652    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:55.943439 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:55.943395    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" event={"ID":"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd","Type":"ContainerStarted","Data":"ef66c2c33008a8451016718f3297a5490d753f371f4f9dec142e653937a1b73b"}
Oct 30 21:54:56.087735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.087691    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:54:56.088896 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.087649    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:56.108122 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.102618    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:56.108122 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.102687    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:56.108122 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.104389    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:56.222590 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.216912    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:56.222590 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.216921    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:56.313608 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.313568    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"]
Oct 30 21:54:56.349494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.322564    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.417497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.397029    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d46316-4736-474f-aa55-71da33c19818-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.621703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.621611    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.625593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.625557    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.650783 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.650738    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.651112 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.651089    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.651293 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.651277    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.651714 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.651689    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.651936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.651918    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d46316-4736-474f-aa55-71da33c19818-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.652185 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.652165    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zz64\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-kube-api-access-7zz64\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:56.652429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.426510    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"]
Oct 30 21:54:56.759414 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.759369    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.138850 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.138819    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.147595 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.147561    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.151603 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.151573    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.166563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.155566    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d46316-4736-474f-aa55-71da33c19818-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.167046 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.167019    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zz64\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-kube-api-access-7zz64\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.254063 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.254027    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d46316-4736-474f-aa55-71da33c19818-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.254319 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.254300    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.263586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.263553    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.263961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.981695    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:57.289495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.983322    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:54:57.289849 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.289810    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:57.290006 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.106243    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:57.290395 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.105985    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:54:57.290780 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.219093    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d46316-4736-474f-aa55-71da33c19818-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.290915 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.220655    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:54:57.291244 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.220515    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:57.318870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.249976    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" event={"ID":"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd","Type":"ContainerStarted","Data":"b7b444e027dc04ac6f5cf34a3d39193f329610f4e1b3f912304327473328b784"}
Oct 30 21:54:57.326568 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.256224    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d46316-4736-474f-aa55-71da33c19818-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.326809 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.106274    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:54:57.328081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:56.786744    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.328081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.256512    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.328081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.256876    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.328081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.272221    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.328081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.161585    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.328081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.239517    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zz64\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-kube-api-access-7zz64\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.333083 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.333015    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:57.371611 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.371544    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podStartSLOduration=6.371522494 podStartE2EDuration="6.371522494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:57.30278419 +0000 UTC m=+915.960880416" watchObservedRunningTime="2026-10-30 21:54:57.371522494 +0000 UTC m=+916.029618746"
Oct 30 21:54:57.578068 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:57.578020    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:54:58.188841 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.183398    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:58.198808 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.198776    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:58.271496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.270086    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:54:58.285422 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.282762    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:58.285422 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.284848    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:58.296403 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.294583    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:54:58.296403 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.294911    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:54:58.296664 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.296549    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:54:58.323498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.318901    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:58.360519 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.334625    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:58.408507 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.336061    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:58.828492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.826056    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:54:58.887800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.887727    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:54:58.901715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:58.901683    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"]
Oct 30 21:54:59.044614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.018897    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:54:59.145451 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.047028    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:54:59.148713 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.148681    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:59.148959 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.148936    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:59.149118 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.149096    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:54:59.316515 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.306513    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:54:59.404806 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.404714    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:54:59.731804 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.731537    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" event={"ID":"f6d46316-4736-474f-aa55-71da33c19818","Type":"ContainerStarted","Data":"8466c75e4a4a449f989552dc33344dcb0a41ad8aaf43d3c00f003c00399f92b9"}
Oct 30 21:54:59.760793 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:54:59.760758    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:55:00.773989 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:00.773947    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" event={"ID":"f6d46316-4736-474f-aa55-71da33c19818","Type":"ContainerStarted","Data":"e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1"}
Oct 30 21:55:00.815224 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:00.815046    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podStartSLOduration=5.814405045 podStartE2EDuration="5.814405045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:00.811994139 +0000 UTC m=+919.470090373" watchObservedRunningTime="2026-10-30 21:55:00.814405045 +0000 UTC m=+919.472501267"
Oct 30 21:55:01.646006 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:01.644004    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:55:01.646006 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:01.644066    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:55:01.690929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:01.690894    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:55:02.622433 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:02.622396    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:55:02.622867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:02.622454    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:55:03.622533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:03.622492    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:55:03.623006 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:03.622555    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:55:04.622074 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:04.622037    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:55:04.622260 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:04.622098    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:55:04.649150 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:55:04.649113    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:55:05.621808 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:05.621776    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:55:05.621984 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:05.621831    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:55:06.622633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:06.622590    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:55:06.623011 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:06.622651    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 21:55:07.829939 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:07.829902    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:55:07.830918 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:07.830893    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:55:07.831521 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:07.831502    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 21:55:14.110496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:14.109609    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:14.135493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:14.129606    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:14.368073 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:14.367961    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:14.766527 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:14.741841    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 21:55:14.833380 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:14.833337    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:55:15.122553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.120563    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:15.122553 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.121378    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:55:15.134500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.131540    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:15.134500 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.132030    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:55:15.227483 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.227421    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:15.347653 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.347603    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:15.376103 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.375999    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:15.388506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.376724    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:55:15.868520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.851168    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:15.914492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:15.908823    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:55:16.144326 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.140224    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:16.186057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.146630    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:16.244099 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.242841    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:55:16.253425 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.251617    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:55:16.362685 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.362623    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:16.364231 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.364158    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:55:16.462440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.462396    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:16.808552 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.770837    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:55:16.808552 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.770913    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:16.863266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:16.862600    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:17.056498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:17.053314    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:55:17.250494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:17.225948    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:17.302563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:17.252066    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:17.389508 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:17.375581    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:55:17.578883 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:17.576632    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:17.895554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:17.867011    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:18.067525 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:18.059457    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:18.265786 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:18.265754    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 21:55:18.286602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:18.284575    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:18.300589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:18.292532    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 21:55:18.523571 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:18.464021    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:18.581530 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:18.577231    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:18.874675 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:18.874573    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:18.941799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:18.941767    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 21:55:19.077518 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.066163    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:55:19.077518 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.075709    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:55:19.234408 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.234042    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:55:19.234408 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.234106    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:19.358229 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.353954    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 21:55:19.385915 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.384673    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:19.426827 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.426798    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 21:55:19.428110 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.428085    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 21:55:19.436094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:19.436072    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 21:55:22.987298 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:22.987257    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"]
Oct 30 21:55:22.995403 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:22.995378    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.025311 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.025231    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"]
Oct 30 21:55:23.125838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.125804    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a21738-5faf-4cea-899c-3be3bc5c62f2-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.126041 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.125852    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.126041 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.125944    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.126041 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.125997    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.126175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.126158    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.126217 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.126189    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjxkw\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-kube-api-access-gjxkw\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.126217 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.126206    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.126325 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.126313    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.126426 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.126409    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.227668 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.227640    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.227817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.227683    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.227859 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.227804    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a21738-5faf-4cea-899c-3be3bc5c62f2-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.227894 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.227862    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.227929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.227912    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.227965 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.227950    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.228105 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.228087    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.228169 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.228119    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.228169 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.228129    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.228280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.228180    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.228280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.228187    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjxkw\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-kube-api-access-gjxkw\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.228280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.228233    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.228680 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.228402    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.228730 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.228705    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a21738-5faf-4cea-899c-3be3bc5c62f2-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.230414 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.230390    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.230934 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.230918    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.239612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.239557    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.239719 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.239702    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjxkw\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-kube-api-access-gjxkw\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.310799 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.310764    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:23.604781 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.604755    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"]
Oct 30 21:55:23.606990 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:55:23.606963    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a21738_5faf_4cea_899c_3be3bc5c62f2.slice/crio-47451860c520b3b0ff37c4f764b6644b91f08b7621b93746fa680b00f2c5b50d WatchSource:0}: Error finding container 47451860c520b3b0ff37c4f764b6644b91f08b7621b93746fa680b00f2c5b50d: Status 404 returned error can't find the container with id 47451860c520b3b0ff37c4f764b6644b91f08b7621b93746fa680b00f2c5b50d
Oct 30 21:55:23.610038 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.610001    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:23.610145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.610068    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:23.610145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:23.610097    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:24.444503 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:24.440350    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" event={"ID":"97a21738-5faf-4cea-899c-3be3bc5c62f2","Type":"ContainerStarted","Data":"46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a"}
Oct 30 21:55:24.445162 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:24.445120    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" event={"ID":"97a21738-5faf-4cea-899c-3be3bc5c62f2","Type":"ContainerStarted","Data":"47451860c520b3b0ff37c4f764b6644b91f08b7621b93746fa680b00f2c5b50d"}
Oct 30 21:55:24.471295 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:24.471235    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" podStartSLOduration=2.471217766 podStartE2EDuration="2.471217766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:24.469070219 +0000 UTC m=+943.127166518" watchObservedRunningTime="2026-10-30 21:55:24.471217766 +0000 UTC m=+943.129313991"
Oct 30 21:55:25.311283 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:25.311255    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:25.312984 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:25.312960    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:55:25.313103 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:25.313009    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:55:26.311192 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:26.311161    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:55:26.311651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:26.311217    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:55:27.311419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:27.311389    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:55:27.311938 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:27.311448    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:55:28.311543 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:28.311506    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:55:28.312022 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:28.311574    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:55:29.312077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:29.312049    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:55:29.312513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:29.312114    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:55:30.311892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.311856    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 21:55:30.312101 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.311919    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 21:55:30.425151 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.425112    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"]
Oct 30 21:55:30.429822 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.429798    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.435549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.435527    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2c9c95-openshift-default-dockercfg-whdd6\""
Oct 30 21:55:30.449867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.449842    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"]
Oct 30 21:55:30.694659 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.694612    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.694999 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.694974    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.695187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.695159    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.695332 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.695310    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jffqm\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-kube-api-access-jffqm\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.695543 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.695525    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.695720 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.695696    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.696144 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.696127    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.696346 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.696323    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.696552 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.696523    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.821490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.818172    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.821745 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.821713    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.821958 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.821937    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.839493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.834250    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.839493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.834420    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.839493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.834959    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.854489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.844566    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.854489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.844643    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.854489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.844676    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jffqm\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-kube-api-access-jffqm\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.854489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.844737    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.854489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.844795    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.854489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.845430    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.854489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.845617    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.854489 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.845892    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.863623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.862781    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.882233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.879361    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jffqm\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-kube-api-access-jffqm\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.889167 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.889088    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:30.890270 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:30.890209    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:31.044804 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.044715    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:31.396544 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.395347    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"]
Oct 30 21:55:31.406848 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:55:31.406812    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec33fd1_e17b_4e07_8b4d_fe1d78a09152.slice/crio-e20f1bc24442d566aa1cec4a128cae6811b811a6dfcc65579495e2b3edb3ea2d WatchSource:0}: Error finding container e20f1bc24442d566aa1cec4a128cae6811b811a6dfcc65579495e2b3edb3ea2d: Status 404 returned error can't find the container with id e20f1bc24442d566aa1cec4a128cae6811b811a6dfcc65579495e2b3edb3ea2d
Oct 30 21:55:31.409265 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.409221    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:31.409417 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.409316    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:31.409417 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.409358    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:31.472129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.472100    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:31.473163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.473136    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:31.476837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.476707    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 21:55:31.803934 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.803898    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" event={"ID":"eec33fd1-e17b-4e07-8b4d-fe1d78a09152","Type":"ContainerStarted","Data":"ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea"}
Oct 30 21:55:31.804135 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.803992    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" event={"ID":"eec33fd1-e17b-4e07-8b4d-fe1d78a09152","Type":"ContainerStarted","Data":"e20f1bc24442d566aa1cec4a128cae6811b811a6dfcc65579495e2b3edb3ea2d"}
Oct 30 21:55:31.828909 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:31.828859    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podStartSLOduration=1.828845794 podStartE2EDuration="1.828845794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:31.825503587 +0000 UTC m=+950.483599807" watchObservedRunningTime="2026-10-30 21:55:31.828845794 +0000 UTC m=+950.486942017"
Oct 30 21:55:32.079938 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:32.076234    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:32.112495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:32.110245    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:32.475842 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:32.475813    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:55:32.810508 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:32.810386    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:32.811099 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:32.811083    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:55:34.740737 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:55:34.740703    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:55:35.993979 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:35.993950    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"]
Oct 30 21:55:35.998597 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:35.998579    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:36.021744 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.021719    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2c9c95\""
Oct 30 21:55:36.024516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.024498    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2c9c95-dockercfg-bhtf2\""
Oct 30 21:55:36.031515 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.031495    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"]
Oct 30 21:55:36.083513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.083449    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2b296f-2b86-4e0b-9271-e659a672fb11-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs\" (UID: \"7b2b296f-2b86-4e0b-9271-e659a672fb11\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:36.085693 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.085660    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqc8w\" (UniqueName: \"kubernetes.io/projected/7b2b296f-2b86-4e0b-9271-e659a672fb11-kube-api-access-hqc8w\") pod \"payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs\" (UID: \"7b2b296f-2b86-4e0b-9271-e659a672fb11\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:36.200397 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.200252    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqc8w\" (UniqueName: \"kubernetes.io/projected/7b2b296f-2b86-4e0b-9271-e659a672fb11-kube-api-access-hqc8w\") pod \"payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs\" (UID: \"7b2b296f-2b86-4e0b-9271-e659a672fb11\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:36.201158 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.201135    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2b296f-2b86-4e0b-9271-e659a672fb11-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs\" (UID: \"7b2b296f-2b86-4e0b-9271-e659a672fb11\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:36.203026 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.202489    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2b296f-2b86-4e0b-9271-e659a672fb11-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs\" (UID: \"7b2b296f-2b86-4e0b-9271-e659a672fb11\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:36.271622 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.267223    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqc8w\" (UniqueName: \"kubernetes.io/projected/7b2b296f-2b86-4e0b-9271-e659a672fb11-kube-api-access-hqc8w\") pod \"payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs\" (UID: \"7b2b296f-2b86-4e0b-9271-e659a672fb11\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:36.313493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.313053    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:36.824504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.824408    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"]
Oct 30 21:55:36.825250 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:55:36.825224    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2b296f_2b86_4e0b_9271_e659a672fb11.slice/crio-a606fd189a7974d59ff08fd6aa1b68a8757e18a8db53d5c5d09fbdd35158cc57 WatchSource:0}: Error finding container a606fd189a7974d59ff08fd6aa1b68a8757e18a8db53d5c5d09fbdd35158cc57: Status 404 returned error can't find the container with id a606fd189a7974d59ff08fd6aa1b68a8757e18a8db53d5c5d09fbdd35158cc57
Oct 30 21:55:36.880499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:36.879033    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs" event={"ID":"7b2b296f-2b86-4e0b-9271-e659a672fb11","Type":"ContainerStarted","Data":"a606fd189a7974d59ff08fd6aa1b68a8757e18a8db53d5c5d09fbdd35158cc57"}
Oct 30 21:55:37.901715 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:37.901585    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs" event={"ID":"7b2b296f-2b86-4e0b-9271-e659a672fb11","Type":"ContainerStarted","Data":"af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db"}
Oct 30 21:55:37.920543 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:37.901963    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:37.926823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:37.926779    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs" podStartSLOduration=2.92676346 podStartE2EDuration="2.92676346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:37.923917362 +0000 UTC m=+956.582013585" watchObservedRunningTime="2026-10-30 21:55:37.92676346 +0000 UTC m=+956.584859683"
Oct 30 21:55:44.076853 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:44.075182    4282 patch_prober.go:29] interesting pod/e2e-mt-689a2563-openshift-default-d86655557-2ktpm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": read tcp 10.133.0.2:58122->10.133.0.74:15021: read: connection reset by peer" start-of-body=
Oct 30 21:55:44.076853 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:44.075260    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": read tcp 10.133.0.2:58122->10.133.0.74:15021: read: connection reset by peer"
Oct 30 21:55:44.974353 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:44.974318    4282 generic.go:356] "Generic (PLEG): container finished" podID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerID="b7b444e027dc04ac6f5cf34a3d39193f329610f4e1b3f912304327473328b784" exitCode=0
Oct 30 21:55:44.974580 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:44.974398    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" event={"ID":"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd","Type":"ContainerDied","Data":"b7b444e027dc04ac6f5cf34a3d39193f329610f4e1b3f912304327473328b784"}
Oct 30 21:55:44.974929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:44.974912    4282 scope.go:122] "RemoveContainer" containerID="b7b444e027dc04ac6f5cf34a3d39193f329610f4e1b3f912304327473328b784"
Oct 30 21:55:44.976521 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:44.976495    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:44.976583 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:44.976560    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:44.976620 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:44.976591    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:55:46.011340 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:46.011294    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" event={"ID":"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd","Type":"ContainerStarted","Data":"f31f67b42b5e2659f32439918c3b0372d6559ae3751f6c992396fa1984774b0c"}
Oct 30 21:55:46.182141 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:46.182107    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:55:46.182329 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:46.182308    4282 patch_prober.go:29] interesting pod/e2e-mt-689a2563-openshift-default-d86655557-2ktpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:55:46.182372 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:46.182357    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:55:47.182931 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:47.182898    4282 patch_prober.go:29] interesting pod/e2e-mt-689a2563-openshift-default-d86655557-2ktpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:55:47.183430 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:47.182953    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:55:48.182259 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:48.182220    4282 patch_prober.go:29] interesting pod/e2e-mt-689a2563-openshift-default-d86655557-2ktpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:55:48.182427 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:48.182274    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:55:48.930492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:48.928063    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:55:49.221734 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:49.221570    4282 patch_prober.go:29] interesting pod/e2e-mt-689a2563-openshift-default-d86655557-2ktpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:55:49.273531 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:49.265351    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:55:50.183494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:50.182831    4282 patch_prober.go:29] interesting pod/e2e-mt-689a2563-openshift-default-d86655557-2ktpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:55:50.183494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:50.182904    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:55:51.182988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:51.182953    4282 patch_prober.go:29] interesting pod/e2e-mt-689a2563-openshift-default-d86655557-2ktpm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:55:51.183172 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:51.183016    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 21:55:52.186147 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:52.186112    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:55:52.186576 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:52.186408    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:55:52.187142 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:55:52.187122    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:56:04.649632 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:56:04.649595    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:56:05.583093 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.583062    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"]
Oct 30 21:56:05.587892 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.587874    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:05.600180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.600160    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-79724b\""
Oct 30 21:56:05.615744 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.615720    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"]
Oct 30 21:56:05.621315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.621286    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"]
Oct 30 21:56:05.623133 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.623115    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-79724b-dockercfg-k5pgk\""
Oct 30 21:56:05.626267 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.626247    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:05.645367 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.645343    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"]
Oct 30 21:56:05.666335 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.666312    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f4f706-a1a4-47c6-b68a-73d76a146583-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb\" (UID: \"30f4f706-a1a4-47c6-b68a-73d76a146583\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:05.666773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.666376    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b857469-db5e-4810-95b0-fd29fe7bbefd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v\" (UID: \"2b857469-db5e-4810-95b0-fd29fe7bbefd\") " pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:05.666773 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.666719    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2g4p\" (UniqueName: \"kubernetes.io/projected/30f4f706-a1a4-47c6-b68a-73d76a146583-kube-api-access-l2g4p\") pod \"payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb\" (UID: \"30f4f706-a1a4-47c6-b68a-73d76a146583\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:05.666875 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.666780    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4x4w\" (UniqueName: \"kubernetes.io/projected/2b857469-db5e-4810-95b0-fd29fe7bbefd-kube-api-access-z4x4w\") pod \"payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v\" (UID: \"2b857469-db5e-4810-95b0-fd29fe7bbefd\") " pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:05.830089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.781915    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4x4w\" (UniqueName: \"kubernetes.io/projected/2b857469-db5e-4810-95b0-fd29fe7bbefd-kube-api-access-z4x4w\") pod \"payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v\" (UID: \"2b857469-db5e-4810-95b0-fd29fe7bbefd\") " pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:05.830628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.830596    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f4f706-a1a4-47c6-b68a-73d76a146583-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb\" (UID: \"30f4f706-a1a4-47c6-b68a-73d76a146583\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:05.830859 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.830842    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b857469-db5e-4810-95b0-fd29fe7bbefd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v\" (UID: \"2b857469-db5e-4810-95b0-fd29fe7bbefd\") " pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:05.831236 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.831219    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2g4p\" (UniqueName: \"kubernetes.io/projected/30f4f706-a1a4-47c6-b68a-73d76a146583-kube-api-access-l2g4p\") pod \"payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb\" (UID: \"30f4f706-a1a4-47c6-b68a-73d76a146583\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:05.881543 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.832757    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f4f706-a1a4-47c6-b68a-73d76a146583-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb\" (UID: \"30f4f706-a1a4-47c6-b68a-73d76a146583\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:05.921665 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.855571    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b857469-db5e-4810-95b0-fd29fe7bbefd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v\" (UID: \"2b857469-db5e-4810-95b0-fd29fe7bbefd\") " pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:06.022493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.946178    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4x4w\" (UniqueName: \"kubernetes.io/projected/2b857469-db5e-4810-95b0-fd29fe7bbefd-kube-api-access-z4x4w\") pod \"payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v\" (UID: \"2b857469-db5e-4810-95b0-fd29fe7bbefd\") " pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:06.022493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:05.946487    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:06.022493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:06.007083    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:56:06.022493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:06.009131    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2g4p\" (UniqueName: \"kubernetes.io/projected/30f4f706-a1a4-47c6-b68a-73d76a146583-kube-api-access-l2g4p\") pod \"payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb\" (UID: \"30f4f706-a1a4-47c6-b68a-73d76a146583\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:06.203539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:06.202760    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:06.522982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:06.514621    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"]
Oct 30 21:56:06.531622 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:06.531599    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"]
Oct 30 21:56:06.535524 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:56:06.535495    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f4f706_a1a4_47c6_b68a_73d76a146583.slice/crio-5ee1ee36e9e5ae3a0cc5ed760f6ebfe11c8e8f60feed4a8ac23029391eff6989 WatchSource:0}: Error finding container 5ee1ee36e9e5ae3a0cc5ed760f6ebfe11c8e8f60feed4a8ac23029391eff6989: Status 404 returned error can't find the container with id 5ee1ee36e9e5ae3a0cc5ed760f6ebfe11c8e8f60feed4a8ac23029391eff6989
Oct 30 21:56:07.509494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:07.502853    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb" event={"ID":"30f4f706-a1a4-47c6-b68a-73d76a146583","Type":"ContainerStarted","Data":"74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140"}
Oct 30 21:56:07.509494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:07.502931    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb" event={"ID":"30f4f706-a1a4-47c6-b68a-73d76a146583","Type":"ContainerStarted","Data":"5ee1ee36e9e5ae3a0cc5ed760f6ebfe11c8e8f60feed4a8ac23029391eff6989"}
Oct 30 21:56:07.509494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:07.504153    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:07.509494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:07.505975    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v" event={"ID":"2b857469-db5e-4810-95b0-fd29fe7bbefd","Type":"ContainerStarted","Data":"faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62"}
Oct 30 21:56:07.509494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:07.506016    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v" event={"ID":"2b857469-db5e-4810-95b0-fd29fe7bbefd","Type":"ContainerStarted","Data":"85bc971522362cbc85681acd22b92daf847ad54c9425536f98c8c4e722e31f94"}
Oct 30 21:56:07.509494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:07.506354    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:07.524959 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:07.524718    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb" podStartSLOduration=2.524704977 podStartE2EDuration="2.524704977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:07.523880104 +0000 UTC m=+986.181976324" watchObservedRunningTime="2026-10-30 21:56:07.524704977 +0000 UTC m=+986.182801199"
Oct 30 21:56:07.545896 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:07.545849    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v" podStartSLOduration=2.545835136 podStartE2EDuration="2.545835136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:07.543913373 +0000 UTC m=+986.202009608" watchObservedRunningTime="2026-10-30 21:56:07.545835136 +0000 UTC m=+986.203931359"
Oct 30 21:56:10.200134 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.200102    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"]
Oct 30 21:56:10.204386 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.204364    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.208661 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.208637    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-79724b-serving-cert\""
Oct 30 21:56:10.224241 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.224215    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"]
Oct 30 21:56:10.271749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.271721    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-maas-api-tls\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.271906 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.271780    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895fd1a-8e26-467b-aee3-f8b5e271993f-openshift-service-ca\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.271906 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.271852    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc24d\" (UniqueName: \"kubernetes.io/projected/3895fd1a-8e26-467b-aee3-f8b5e271993f-kube-api-access-mc24d\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.271906 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.271898    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-metrics-certs\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.379496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.377427    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-metrics-certs\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.379496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.377745    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-maas-api-tls\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.379496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.377815    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895fd1a-8e26-467b-aee3-f8b5e271993f-openshift-service-ca\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.379496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.377857    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc24d\" (UniqueName: \"kubernetes.io/projected/3895fd1a-8e26-467b-aee3-f8b5e271993f-kube-api-access-mc24d\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.379854 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.379765    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895fd1a-8e26-467b-aee3-f8b5e271993f-openshift-service-ca\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.391672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.391627    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-metrics-certs\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.417089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.417010    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-maas-api-tls\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.419679 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.419650    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc24d\" (UniqueName: \"kubernetes.io/projected/3895fd1a-8e26-467b-aee3-f8b5e271993f-kube-api-access-mc24d\") pod \"maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.526110 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.526027    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:10.898946 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.887205    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"]
Oct 30 21:56:10.905245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:10.904532    4282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:56:11.309767 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.309732    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"]
Oct 30 21:56:11.315232 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.315211    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.337289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.337260    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"]
Oct 30 21:56:11.398697 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.398661    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.399018 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.398997    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.399335 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.399312    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhwp\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-kube-api-access-xbhwp\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.399556 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.399528    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.399650 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.399570    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.399650 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.399640    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.399729 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.399683    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.399729 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.399707    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940dbb6d-1424-4a15-89df-a9ec9d77534d-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.399879 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.399862    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.500844 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.500818    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.500962 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.500877    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.501038 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501018    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhwp\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-kube-api-access-xbhwp\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.501095 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501062    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.501148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501121    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.501226 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501206    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.501226 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501220    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.501380 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501275    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.501380 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501302    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940dbb6d-1424-4a15-89df-a9ec9d77534d-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.501529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501459    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.502019 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.501839    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.502381 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.502352    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.502534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.502516    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940dbb6d-1424-4a15-89df-a9ec9d77534d-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.502713 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.502688    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.503718 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.503698    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.504409 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.504389    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.516025 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.515998    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.517197 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.517170    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhwp\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-kube-api-access-xbhwp\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:11.533375 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.533341    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x" event={"ID":"3895fd1a-8e26-467b-aee3-f8b5e271993f","Type":"ContainerStarted","Data":"6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc"}
Oct 30 21:56:11.533539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.533395    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x" event={"ID":"3895fd1a-8e26-467b-aee3-f8b5e271993f","Type":"ContainerStarted","Data":"b5649276c75d404c48f8733f44b9a5233c5795b03186fe3613c9d733e7cdfc7d"}
Oct 30 21:56:11.533539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.533440    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:11.553765 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.553706    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x" podStartSLOduration=1.029516315 podStartE2EDuration="1.553687111s" totalImagesPullingTime="524.170796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:11.551568493 +0000 UTC m=+990.209664719" watchObservedRunningTime="2026-10-30 21:56:11.553687111 +0000 UTC m=+990.211783335"
Oct 30 21:56:11.633647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:11.633578    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:12.070382 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:12.069304    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"]
Oct 30 21:56:12.073538 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:56:12.072844    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940dbb6d_1424_4a15_89df_a9ec9d77534d.slice/crio-00f924ecebeb129722546c25f02f54b1386ac4526ee8ee093c02dec1b236e1df WatchSource:0}: Error finding container 00f924ecebeb129722546c25f02f54b1386ac4526ee8ee093c02dec1b236e1df: Status 404 returned error can't find the container with id 00f924ecebeb129722546c25f02f54b1386ac4526ee8ee093c02dec1b236e1df
Oct 30 21:56:12.076391 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:12.076358    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:12.076541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:12.076432    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:12.076541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:12.076489    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:12.540978 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:12.540943    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" event={"ID":"940dbb6d-1424-4a15-89df-a9ec9d77534d","Type":"ContainerStarted","Data":"bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21"}
Oct 30 21:56:12.541529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:12.541005    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" event={"ID":"940dbb6d-1424-4a15-89df-a9ec9d77534d","Type":"ContainerStarted","Data":"00f924ecebeb129722546c25f02f54b1386ac4526ee8ee093c02dec1b236e1df"}
Oct 30 21:56:12.572688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:12.572633    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podStartSLOduration=1.572613941 podStartE2EDuration="1.572613941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:12.569739729 +0000 UTC m=+991.227835983" watchObservedRunningTime="2026-10-30 21:56:12.572613941 +0000 UTC m=+991.230710167"
Oct 30 21:56:12.634216 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:12.634194    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:13.644490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:13.643613    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:56:13.644490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:13.643674    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:56:14.636762 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:14.635924    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:56:14.636762 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:14.635981    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:56:15.634051 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:15.634016    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:56:15.634589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:15.634073    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:56:16.649502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:16.645221    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:56:16.649502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:16.645293    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:56:17.547107 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:17.547080    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:56:17.634757 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:17.634728    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:56:17.634915 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:17.634785    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:56:18.515294 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:18.515267    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:56:18.515726 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:18.515371    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:56:18.633982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:18.633952    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 21:56:18.634170 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:18.634009    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 21:56:19.914064 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:19.913995    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:19.914817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:19.914791    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:19.915947 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:19.915924    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 21:56:34.650217 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:56:34.650182    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:56:35.995922 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:35.995873    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"]
Oct 30 21:56:38.116530 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:38.116496    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:56:40.996577 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:40.996535    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs" podUID="7b2b296f-2b86-4e0b-9271-e659a672fb11" containerName="payload-pre-processing" containerID="cri-o://af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db" gracePeriod=25
Oct 30 21:56:41.345456 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.345421    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:56:41.397626 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.397593    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2b296f-2b86-4e0b-9271-e659a672fb11-kube-api-access-hqc8w\" (UniqueName: \"kubernetes.io/projected/7b2b296f-2b86-4e0b-9271-e659a672fb11-kube-api-access-hqc8w\") pod \"7b2b296f-2b86-4e0b-9271-e659a672fb11\" (UID: \"7b2b296f-2b86-4e0b-9271-e659a672fb11\") "
Oct 30 21:56:41.397913 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.397890    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2b296f-2b86-4e0b-9271-e659a672fb11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2b296f-2b86-4e0b-9271-e659a672fb11-plugins-config-volume\") pod \"7b2b296f-2b86-4e0b-9271-e659a672fb11\" (UID: \"7b2b296f-2b86-4e0b-9271-e659a672fb11\") "
Oct 30 21:56:41.398222 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.398201    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2b296f-2b86-4e0b-9271-e659a672fb11-plugins-config-volume" pod "7b2b296f-2b86-4e0b-9271-e659a672fb11" (UID: "7b2b296f-2b86-4e0b-9271-e659a672fb11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:41.399025 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.399001    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b2b296f-2b86-4e0b-9271-e659a672fb11-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:41.399817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.399798    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2b296f-2b86-4e0b-9271-e659a672fb11-kube-api-access-hqc8w" pod "7b2b296f-2b86-4e0b-9271-e659a672fb11" (UID: "7b2b296f-2b86-4e0b-9271-e659a672fb11"). InnerVolumeSpecName "kube-api-access-hqc8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:41.500598 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.500574    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqc8w\" (UniqueName: \"kubernetes.io/projected/7b2b296f-2b86-4e0b-9271-e659a672fb11-kube-api-access-hqc8w\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:56:41.937955 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.937921    4282 generic.go:356] "Generic (PLEG): container finished" podID="7b2b296f-2b86-4e0b-9271-e659a672fb11" containerID="af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db" exitCode=0
Oct 30 21:56:41.938129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.937967    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"
Oct 30 21:56:41.938129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.938005    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs" event={"ID":"7b2b296f-2b86-4e0b-9271-e659a672fb11","Type":"ContainerDied","Data":"af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db"}
Oct 30 21:56:41.938129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.938061    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs" event={"ID":"7b2b296f-2b86-4e0b-9271-e659a672fb11","Type":"ContainerDied","Data":"a606fd189a7974d59ff08fd6aa1b68a8757e18a8db53d5c5d09fbdd35158cc57"}
Oct 30 21:56:41.938129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.938082    4282 scope.go:122] "RemoveContainer" containerID="af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db"
Oct 30 21:56:41.950271 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.950246    4282 scope.go:122] "RemoveContainer" containerID="af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db"
Oct 30 21:56:41.950598 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:56:41.950575    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db\": container with ID starting with af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db not found: ID does not exist" containerID="af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db"
Oct 30 21:56:41.950699 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.950609    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db"} err="failed to get container status \"af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db\": rpc error: code = NotFound desc = could not find container \"af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db\": container with ID starting with af3da4ddf7caead5e0f9cfed4af0d4c953aee3cf5f993a90acc09830dbc462db not found: ID does not exist"
Oct 30 21:56:41.979963 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.979929    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"]
Oct 30 21:56:41.983164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:41.983136    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2c9c95-6b4dcbd79-796xs"]
Oct 30 21:56:43.491847 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:43.491811    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2b296f-2b86-4e0b-9271-e659a672fb11" path="/var/lib/kubelet/pods/7b2b296f-2b86-4e0b-9271-e659a672fb11/volumes"
Oct 30 21:56:45.666097 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.666059    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"]
Oct 30 21:56:45.667012 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.666984    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7b2b296f-2b86-4e0b-9271-e659a672fb11" containerName="payload-pre-processing"
Oct 30 21:56:45.667012 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.667010    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2b296f-2b86-4e0b-9271-e659a672fb11" containerName="payload-pre-processing"
Oct 30 21:56:45.667242 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.667226    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2b296f-2b86-4e0b-9271-e659a672fb11" containerName="payload-pre-processing"
Oct 30 21:56:45.673020 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.672996    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.756492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.748493    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"]
Oct 30 21:56:45.802047 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.801996    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.802268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.802253    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.802400 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.802391    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117522f-b5f2-45df-a655-076a4ec35a6c-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.802498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.802485    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.812277 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.802712    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.812607 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.812583    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.814419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.814337    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cslb\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-kube-api-access-5cslb\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.822780 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.822756    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.823407 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.823382    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.925726 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.925688    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.925880 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.925749    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.925880 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.925838    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117522f-b5f2-45df-a655-076a4ec35a6c-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.925880 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.925872    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926043 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.925923    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926043 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.925971    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926043 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.926009    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cslb\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-kube-api-access-5cslb\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.926057    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.926127    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.926265    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.926413    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926645 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.926625    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117522f-b5f2-45df-a655-076a4ec35a6c-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.926713    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.926800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.926778    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.928231 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.928208    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.928746 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.928722    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.963279 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.963257    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:45.964910 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:45.964890    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cslb\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-kube-api-access-5cslb\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:46.052029 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:46.051999    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:46.353576 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:46.353550    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"]
Oct 30 21:56:46.354337 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:56:46.354310    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3117522f_b5f2_45df_a655_076a4ec35a6c.slice/crio-188727ad8a601497afa97f21fd36fb7ae672dfcc682c0db11f17f2a7338f71c6 WatchSource:0}: Error finding container 188727ad8a601497afa97f21fd36fb7ae672dfcc682c0db11f17f2a7338f71c6: Status 404 returned error can't find the container with id 188727ad8a601497afa97f21fd36fb7ae672dfcc682c0db11f17f2a7338f71c6
Oct 30 21:56:46.357812 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:46.357778    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:46.357920 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:46.357875    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:46.357988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:46.357924    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:47.067960 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.067927    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" event={"ID":"3117522f-b5f2-45df-a655-076a4ec35a6c","Type":"ContainerStarted","Data":"b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f"}
Oct 30 21:56:47.068361 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.067986    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" event={"ID":"3117522f-b5f2-45df-a655-076a4ec35a6c","Type":"ContainerStarted","Data":"188727ad8a601497afa97f21fd36fb7ae672dfcc682c0db11f17f2a7338f71c6"}
Oct 30 21:56:47.104981 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.104936    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" podStartSLOduration=2.104923114 podStartE2EDuration="2.104923114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:47.103805727 +0000 UTC m=+1025.761901951" watchObservedRunningTime="2026-10-30 21:56:47.104923114 +0000 UTC m=+1025.763019443"
Oct 30 21:56:47.878491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.877625    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:56:47.878491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.877697    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:56:47.950685 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.950108    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"]
Oct 30 21:56:47.976176 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.976142    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"]
Oct 30 21:56:47.977746 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.977723    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"]
Oct 30 21:56:47.980883 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:47.980861    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:48.020699 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.020669    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-03f0a0-dockercfg-k6k8g\""
Oct 30 21:56:48.040728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.040679    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-03f0a0\""
Oct 30 21:56:48.047731 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.047701    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"]
Oct 30 21:56:48.048066 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.048045    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:48.052180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.052160    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:48.053196 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.053174    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:56:48.053333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.053310    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:56:48.085896 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.082523    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0015e3-f2b6-4d37-9f3c-b183498228fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2\" (UID: \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:48.089063 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.086424    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgc5p\" (UniqueName: \"kubernetes.io/projected/0d0015e3-f2b6-4d37-9f3c-b183498228fd-kube-api-access-fgc5p\") pod \"payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2\" (UID: \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:48.098545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.098491    4282 generic.go:356] "Generic (PLEG): container finished" podID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerID="ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea" exitCode=0
Oct 30 21:56:48.098687 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.098554    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" event={"ID":"eec33fd1-e17b-4e07-8b4d-fe1d78a09152","Type":"ContainerDied","Data":"ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea"}
Oct 30 21:56:48.099144 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.099121    4282 scope.go:122] "RemoveContainer" containerID="ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea"
Oct 30 21:56:48.100939 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.100905    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:48.101031 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.101017    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:48.101105 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.101085    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:56:48.198654 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.188371    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgc5p\" (UniqueName: \"kubernetes.io/projected/0d0015e3-f2b6-4d37-9f3c-b183498228fd-kube-api-access-fgc5p\") pod \"payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2\" (UID: \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:48.198654 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.188579    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0015e3-f2b6-4d37-9f3c-b183498228fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2\" (UID: \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:48.198654 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.188832    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qmc5\" (UniqueName: \"kubernetes.io/projected/08a29d12-2b5f-43a4-a00b-07e87430976b-kube-api-access-7qmc5\") pod \"payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285\" (UID: \"08a29d12-2b5f-43a4-a00b-07e87430976b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:48.198654 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.188975    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a29d12-2b5f-43a4-a00b-07e87430976b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285\" (UID: \"08a29d12-2b5f-43a4-a00b-07e87430976b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:48.203089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.199797    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0015e3-f2b6-4d37-9f3c-b183498228fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2\" (UID: \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:48.204587 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.204565    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgc5p\" (UniqueName: \"kubernetes.io/projected/0d0015e3-f2b6-4d37-9f3c-b183498228fd-kube-api-access-fgc5p\") pod \"payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2\" (UID: \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:48.302492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.295209    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qmc5\" (UniqueName: \"kubernetes.io/projected/08a29d12-2b5f-43a4-a00b-07e87430976b-kube-api-access-7qmc5\") pod \"payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285\" (UID: \"08a29d12-2b5f-43a4-a00b-07e87430976b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:48.302492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.295325    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a29d12-2b5f-43a4-a00b-07e87430976b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285\" (UID: \"08a29d12-2b5f-43a4-a00b-07e87430976b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:48.302492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.296312    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a29d12-2b5f-43a4-a00b-07e87430976b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285\" (UID: \"08a29d12-2b5f-43a4-a00b-07e87430976b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:48.378016 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.373643    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qmc5\" (UniqueName: \"kubernetes.io/projected/08a29d12-2b5f-43a4-a00b-07e87430976b-kube-api-access-7qmc5\") pod \"payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285\" (UID: \"08a29d12-2b5f-43a4-a00b-07e87430976b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:48.378016 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.373957    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:48.399686 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.398689    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:48.817565 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.817539    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"]
Oct 30 21:56:48.818161 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:56:48.818138    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0015e3_f2b6_4d37_9f3c_b183498228fd.slice/crio-6d656564b9cc71a9328690cda28441ef0057686c78906810677e85ce6b4a568c WatchSource:0}: Error finding container 6d656564b9cc71a9328690cda28441ef0057686c78906810677e85ce6b4a568c: Status 404 returned error can't find the container with id 6d656564b9cc71a9328690cda28441ef0057686c78906810677e85ce6b4a568c
Oct 30 21:56:48.823570 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:48.823540    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"]
Oct 30 21:56:48.824482 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:56:48.824438    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a29d12_2b5f_43a4_a00b_07e87430976b.slice/crio-9c6112c11302ef72e1609abd5b115614b2e512bed096c772a418a7e6d2d22c93 WatchSource:0}: Error finding container 9c6112c11302ef72e1609abd5b115614b2e512bed096c772a418a7e6d2d22c93: Status 404 returned error can't find the container with id 9c6112c11302ef72e1609abd5b115614b2e512bed096c772a418a7e6d2d22c93
Oct 30 21:56:49.058492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.057031    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:56:49.058492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.057086    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:56:49.110436 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.110347    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285" event={"ID":"08a29d12-2b5f-43a4-a00b-07e87430976b","Type":"ContainerStarted","Data":"4e6d275495bf48c6d95f3410cabb154711da2dce9ed491f7247c6899a3b4a624"}
Oct 30 21:56:49.110824 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.110439    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285" event={"ID":"08a29d12-2b5f-43a4-a00b-07e87430976b","Type":"ContainerStarted","Data":"9c6112c11302ef72e1609abd5b115614b2e512bed096c772a418a7e6d2d22c93"}
Oct 30 21:56:49.110824 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.110499    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:56:49.112403 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.112374    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2" event={"ID":"0d0015e3-f2b6-4d37-9f3c-b183498228fd","Type":"ContainerStarted","Data":"ee9f1bee0a58a4a254b67b12989e177c148ffa486bd068fce1ff38757ccc5517"}
Oct 30 21:56:49.112534 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.112428    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2" event={"ID":"0d0015e3-f2b6-4d37-9f3c-b183498228fd","Type":"ContainerStarted","Data":"6d656564b9cc71a9328690cda28441ef0057686c78906810677e85ce6b4a568c"}
Oct 30 21:56:49.112765 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.112749    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:56:49.115395 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.115374    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" event={"ID":"eec33fd1-e17b-4e07-8b4d-fe1d78a09152","Type":"ContainerStarted","Data":"842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf"}
Oct 30 21:56:49.129519 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.129455    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285" podStartSLOduration=2.129439271 podStartE2EDuration="2.129439271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:49.128048959 +0000 UTC m=+1027.786145183" watchObservedRunningTime="2026-10-30 21:56:49.129439271 +0000 UTC m=+1027.787535494"
Oct 30 21:56:49.184695 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:49.184643    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2" podStartSLOduration=2.184629625 podStartE2EDuration="2.184629625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:49.180885162 +0000 UTC m=+1027.838981388" watchObservedRunningTime="2026-10-30 21:56:49.184629625 +0000 UTC m=+1027.842725847"
Oct 30 21:56:50.054490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:50.045491    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:56:50.054490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:50.045806    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:56:50.054490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:50.045853    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:56:50.068492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:50.066910    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:56:50.068492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:50.066985    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:56:51.045730 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:51.045694    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:56:51.046145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:51.045746    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:56:51.055833 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:51.055801    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:56:51.056085 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:51.056059    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:56:52.046241 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:52.045901    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:56:52.046241 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:52.045947    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:56:52.057811 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:52.057778    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:56:52.058078 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:52.058051    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:56:53.046559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.045764    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:56:53.046559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.045844    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:56:53.126803 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.126767    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:56:53.126998 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.126825    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 21:56:53.215633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.215601    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"]
Oct 30 21:56:53.229926 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.229895    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"]
Oct 30 21:56:53.230119 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.230075    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:53.239800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.239776    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-689a2563-dockercfg-fwpqw\""
Oct 30 21:56:53.240568 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.240545    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-689a2563\""
Oct 30 21:56:53.332116 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.332034    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc\" (UID: \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:53.332310 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.332155    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzg9q\" (UniqueName: \"kubernetes.io/projected/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-kube-api-access-pzg9q\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc\" (UID: \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:53.434273 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.434231    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzg9q\" (UniqueName: \"kubernetes.io/projected/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-kube-api-access-pzg9q\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc\" (UID: \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:53.434561 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.434541    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc\" (UID: \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:53.435538 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.435508    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc\" (UID: \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:53.490100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.490069    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzg9q\" (UniqueName: \"kubernetes.io/projected/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-kube-api-access-pzg9q\") pod \"payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc\" (UID: \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:53.576498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:53.575945    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:54.052659 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:54.048831    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:56:54.052659 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:54.048892    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:56:54.362453 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:54.350045    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"]
Oct 30 21:56:54.637380 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:54.637318    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:54.637989 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:54.637966    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:54.639336 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:54.639318    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:56:55.058533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:55.054801    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:56:55.058533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:55.054863    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:56:55.413110 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:55.412992    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc" event={"ID":"c42d6a09-5bf9-4e10-8576-44ad7442e7f0","Type":"ContainerStarted","Data":"c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed"}
Oct 30 21:56:55.413110 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:55.413086    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc" event={"ID":"c42d6a09-5bf9-4e10-8576-44ad7442e7f0","Type":"ContainerStarted","Data":"d10baeb8658c98b443062baa4021d770c7f4d6255deb721c5b7c0c17558e59f7"}
Oct 30 21:56:55.413333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:55.413282    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:56:56.059498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:56.045819    4282 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 21:56:56.059498 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:56.045891    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 21:56:57.049800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:57.049771    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:56:57.050215 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:57.050180    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:56:57.051024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:57.051006    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:56:57.078818 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:57.078759    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc" podStartSLOduration=4.078742638 podStartE2EDuration="4.078742638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:55.478441391 +0000 UTC m=+1034.136537615" watchObservedRunningTime="2026-10-30 21:56:57.078742638 +0000 UTC m=+1035.736838864"
Oct 30 21:56:58.037512 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:58.036564    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"]
Oct 30 21:56:58.042571 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:58.042447    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"]
Oct 30 21:56:58.046617 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:58.042671    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy" containerID="cri-o://b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f" gracePeriod=30
Oct 30 21:56:58.310197 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:58.309279    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:58.649638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:58.621022    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:58.694536 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:58.690571    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:58.974571 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:58.974532    4282 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" secret="" err="secret \"e2e-api-a-2c9c95-openshift-default-dockercfg-whdd6\" not found"
Oct 30 21:56:59.879587 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:56:59.877911    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:00.082488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:00.082404    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy" containerID="cri-o://842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf" gracePeriod=30
Oct 30 21:57:00.195613 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:00.195558    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:00.286495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:00.261896    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:57:00.286495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:00.272687    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:57:03.410432 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.410407    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.560336    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-socket\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.560395    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-envoy\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.560562    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-credential-socket\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.560623    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-kube-api-access-5cslb\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-kube-api-access-5cslb\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.560658    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-certs\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.560706    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-token\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-token\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.560842    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-podinfo\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.560991    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3117522f-b5f2-45df-a655-076a4ec35a6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117522f-b5f2-45df-a655-076a4ec35a6c-istiod-ca-cert\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.562137 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.561027    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-data\") pod \"3117522f-b5f2-45df-a655-076a4ec35a6c\" (UID: \"3117522f-b5f2-45df-a655-076a4ec35a6c\") "
Oct 30 21:57:03.563319 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.563170    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-data" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.564229 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.563854    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-socket" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.564652 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.564631    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-certs" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.564949 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.564930    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-credential-socket" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.565925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.565892    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3117522f-b5f2-45df-a655-076a4ec35a6c-istiod-ca-cert" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:03.620902 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.620664    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-kube-api-access-5cslb" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "kube-api-access-5cslb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:03.620902 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.620753    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-podinfo" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:03.620902 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.620803    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-envoy" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:03.620902 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.620857    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-token" pod "3117522f-b5f2-45df-a655-076a4ec35a6c" (UID: "3117522f-b5f2-45df-a655-076a4ec35a6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:03.663614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663585    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.663614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663615    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3117522f-b5f2-45df-a655-076a4ec35a6c-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.663820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663630    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.663820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663644    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.663820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663658    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.663820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663672    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.663820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663686    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cslb\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-kube-api-access-5cslb\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.663820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663702    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3117522f-b5f2-45df-a655-076a4ec35a6c-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:03.663820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:03.663714    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3117522f-b5f2-45df-a655-076a4ec35a6c-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:04.125453 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.125417    4282 generic.go:356] "Generic (PLEG): container finished" podID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerID="b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f" exitCode=0
Oct 30 21:57:04.125672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.125506    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"
Oct 30 21:57:04.125672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.125505    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" event={"ID":"3117522f-b5f2-45df-a655-076a4ec35a6c","Type":"ContainerDied","Data":"b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f"}
Oct 30 21:57:04.125672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.125625    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k" event={"ID":"3117522f-b5f2-45df-a655-076a4ec35a6c","Type":"ContainerDied","Data":"188727ad8a601497afa97f21fd36fb7ae672dfcc682c0db11f17f2a7338f71c6"}
Oct 30 21:57:04.125672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.125650    4282 scope.go:122] "RemoveContainer" containerID="b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f"
Oct 30 21:57:04.136051 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.136033    4282 scope.go:122] "RemoveContainer" containerID="b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f"
Oct 30 21:57:04.136317 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:04.136298    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f\": container with ID starting with b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f not found: ID does not exist" containerID="b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f"
Oct 30 21:57:04.136389 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.136327    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f"} err="failed to get container status \"b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f\": rpc error: code = NotFound desc = could not find container \"b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f\": container with ID starting with b5b5ee221be6acadf0ded0e504b86909fd495eaec9056471f9ba5ef90e32239f not found: ID does not exist"
Oct 30 21:57:04.148775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.148753    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"]
Oct 30 21:57:04.154396 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:04.154378    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-sst2k"]
Oct 30 21:57:04.649024 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:04.648992    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:57:05.550830 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.550801    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" path="/var/lib/kubelet/pods/3117522f-b5f2-45df-a655-076a4ec35a6c/volumes"
Oct 30 21:57:05.569584 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.569562    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:57:05.620730 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.620695    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-certs\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.620878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.620833    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-credential-socket\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.620878 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.620869    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-podinfo\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.620985 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.620955    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-socket\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.621311 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.621023    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-certs" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.621421 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.621073    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-token\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-token\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.621421 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.621152    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-credential-socket" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.621421 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.621273    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-socket" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.621600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.621419    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-envoy\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.621600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.621551    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istiod-ca-cert\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.621600 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.621590    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-data\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.621913 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.621658    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-kube-api-access-jffqm\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-kube-api-access-jffqm\") pod \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\" (UID: \"eec33fd1-e17b-4e07-8b4d-fe1d78a09152\") "
Oct 30 21:57:05.622652 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.622627    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.622774 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.622655    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.622774 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.622671    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.623348 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.623319    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-podinfo" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:57:05.623451 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.623346    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istiod-ca-cert" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:05.623741 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.623721    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-data" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.623952 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.623923    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-envoy" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:57:05.625541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.625514    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-kube-api-access-jffqm" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "kube-api-access-jffqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:05.626042 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.626016    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-token" pod "eec33fd1-e17b-4e07-8b4d-fe1d78a09152" (UID: "eec33fd1-e17b-4e07-8b4d-fe1d78a09152"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:05.729289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.729253    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.729289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.729279    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.729289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.729288    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.729289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.729296    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.729911 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.729305    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:05.729911 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:05.729314    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jffqm\" (UniqueName: \"kubernetes.io/projected/eec33fd1-e17b-4e07-8b4d-fe1d78a09152-kube-api-access-jffqm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:06.160818 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.160781    4282 generic.go:356] "Generic (PLEG): container finished" podID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerID="842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf" exitCode=0
Oct 30 21:57:06.161250 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.161221    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" event={"ID":"eec33fd1-e17b-4e07-8b4d-fe1d78a09152","Type":"ContainerDied","Data":"842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf"}
Oct 30 21:57:06.161506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.161486    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr" event={"ID":"eec33fd1-e17b-4e07-8b4d-fe1d78a09152","Type":"ContainerDied","Data":"e20f1bc24442d566aa1cec4a128cae6811b811a6dfcc65579495e2b3edb3ea2d"}
Oct 30 21:57:06.161701 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.161668    4282 scope.go:122] "RemoveContainer" containerID="842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf"
Oct 30 21:57:06.161993 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.161977    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"
Oct 30 21:57:06.212697 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.210991    4282 scope.go:122] "RemoveContainer" containerID="ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea"
Oct 30 21:57:06.223267 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.220863    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"]
Oct 30 21:57:06.228249 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.227235    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-jfdnr"]
Oct 30 21:57:06.232690 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.232653    4282 scope.go:122] "RemoveContainer" containerID="842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf"
Oct 30 21:57:06.233300 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:06.233278    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf\": container with ID starting with 842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf not found: ID does not exist" containerID="842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf"
Oct 30 21:57:06.233459 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.233432    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf"} err="failed to get container status \"842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf\": rpc error: code = NotFound desc = could not find container \"842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf\": container with ID starting with 842c8f86fa32d5fcc8b3e5983ecca4896bc6c101a8b801f16a99c07d0bdd65bf not found: ID does not exist"
Oct 30 21:57:06.233633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.233618    4282 scope.go:122] "RemoveContainer" containerID="ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea"
Oct 30 21:57:06.234106 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:06.234075    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea\": container with ID starting with ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea not found: ID does not exist" containerID="ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea"
Oct 30 21:57:06.234215 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.234108    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea"} err="failed to get container status \"ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea\": rpc error: code = NotFound desc = could not find container \"ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea\": container with ID starting with ef7b11f09c907e291245c190b546bf92d61e72d3156d90d7eff5a87f09ead7ea not found: ID does not exist"
Oct 30 21:57:06.368756 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.368728    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"]
Oct 30 21:57:06.369349 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369336    4282 cpu_manager.go:414] "containerMap: removing container" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy"
Oct 30 21:57:06.369400 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369351    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy"
Oct 30 21:57:06.369400 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369380    4282 cpu_manager.go:414] "containerMap: removing container" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy"
Oct 30 21:57:06.369400 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369385    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy"
Oct 30 21:57:06.369520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369401    4282 cpu_manager.go:414] "containerMap: removing container" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy"
Oct 30 21:57:06.369520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369406    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy"
Oct 30 21:57:06.369520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369507    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3117522f-b5f2-45df-a655-076a4ec35a6c" containerName="istio-proxy"
Oct 30 21:57:06.369520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369515    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy"
Oct 30 21:57:06.369635 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.369524    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" containerName="istio-proxy"
Oct 30 21:57:06.374318 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.374301    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:06.379233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.379214    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"]
Oct 30 21:57:06.379821 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.379584    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ee0c56\""
Oct 30 21:57:06.384116 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.384099    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:06.384709 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.384690    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"]
Oct 30 21:57:06.407415 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.407388    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"]
Oct 30 21:57:06.454493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.442696    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9qx\" (UniqueName: \"kubernetes.io/projected/9a7d7e86-f143-4876-a5a5-2175ec447a33-kube-api-access-bv9qx\") pod \"payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq\" (UID: \"9a7d7e86-f143-4876-a5a5-2175ec447a33\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:06.454493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.442757    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c229a2-f5f0-49d7-8812-6165d0efb27b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277\" (UID: \"12c229a2-f5f0-49d7-8812-6165d0efb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:06.454493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.442804    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwskq\" (UniqueName: \"kubernetes.io/projected/12c229a2-f5f0-49d7-8812-6165d0efb27b-kube-api-access-xwskq\") pod \"payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277\" (UID: \"12c229a2-f5f0-49d7-8812-6165d0efb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:06.454493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.443046    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d7e86-f143-4876-a5a5-2175ec447a33-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq\" (UID: \"9a7d7e86-f143-4876-a5a5-2175ec447a33\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:06.490493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.483368    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ee0c56-dockercfg-d7hwg\""
Oct 30 21:57:06.522094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.490847    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:57:06.570492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.569534    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d7e86-f143-4876-a5a5-2175ec447a33-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq\" (UID: \"9a7d7e86-f143-4876-a5a5-2175ec447a33\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:06.570492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.569816    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9qx\" (UniqueName: \"kubernetes.io/projected/9a7d7e86-f143-4876-a5a5-2175ec447a33-kube-api-access-bv9qx\") pod \"payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq\" (UID: \"9a7d7e86-f143-4876-a5a5-2175ec447a33\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:06.570492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.569878    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c229a2-f5f0-49d7-8812-6165d0efb27b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277\" (UID: \"12c229a2-f5f0-49d7-8812-6165d0efb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:06.570492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.569925    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwskq\" (UniqueName: \"kubernetes.io/projected/12c229a2-f5f0-49d7-8812-6165d0efb27b-kube-api-access-xwskq\") pod \"payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277\" (UID: \"12c229a2-f5f0-49d7-8812-6165d0efb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:06.578492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.571448    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d7e86-f143-4876-a5a5-2175ec447a33-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq\" (UID: \"9a7d7e86-f143-4876-a5a5-2175ec447a33\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:06.600492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.591640    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c229a2-f5f0-49d7-8812-6165d0efb27b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277\" (UID: \"12c229a2-f5f0-49d7-8812-6165d0efb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:06.653496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.640065    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwskq\" (UniqueName: \"kubernetes.io/projected/12c229a2-f5f0-49d7-8812-6165d0efb27b-kube-api-access-xwskq\") pod \"payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277\" (UID: \"12c229a2-f5f0-49d7-8812-6165d0efb27b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:06.709520 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.709361    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:06.748538 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:06.748509    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9qx\" (UniqueName: \"kubernetes.io/projected/9a7d7e86-f143-4876-a5a5-2175ec447a33-kube-api-access-bv9qx\") pod \"payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq\" (UID: \"9a7d7e86-f143-4876-a5a5-2175ec447a33\") " pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:07.014850 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.014765    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:07.029689 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.029664    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"]
Oct 30 21:57:07.038507 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.035499    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.040823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.040801    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-56c0af-openshift-default-dockercfg-zl986\""
Oct 30 21:57:07.055504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.055480    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"]
Oct 30 21:57:07.087514 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.087399    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-credential-socket\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.087514 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.087485    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-certs\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.087728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.087576    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-envoy\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.087728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.087619    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f390701-2496-414e-ada8-90201df827ae-istio-podinfo\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.087838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.087820    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-socket\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.087888 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.087870    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f390701-2496-414e-ada8-90201df827ae-istiod-ca-cert\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.087995 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.087983    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-istio-token\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.088030 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.088004    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh82j\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-kube-api-access-gh82j\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.088030 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.088024    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-data\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.151599 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.151562    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"]
Oct 30 21:57:07.155898 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:57:07.155868    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c229a2_f5f0_49d7_8812_6165d0efb27b.slice/crio-d24b7543a1d5b344019030b0d2e68bfcd6b95e6003679c4419bc9deaf615efec WatchSource:0}: Error finding container d24b7543a1d5b344019030b0d2e68bfcd6b95e6003679c4419bc9deaf615efec: Status 404 returned error can't find the container with id d24b7543a1d5b344019030b0d2e68bfcd6b95e6003679c4419bc9deaf615efec
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220220    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-credential-socket\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220308    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-certs\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220362    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-envoy\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220399    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f390701-2496-414e-ada8-90201df827ae-istio-podinfo\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220577    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-socket\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220640    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f390701-2496-414e-ada8-90201df827ae-istiod-ca-cert\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220729    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-istio-token\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220759    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh82j\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-kube-api-access-gh82j\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.220785    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-data\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.221250    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-data\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.221806    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-credential-socket\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.253585 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.222084    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-certs\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.289551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.255761    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f390701-2496-414e-ada8-90201df827ae-istiod-ca-cert\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.289551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.261907    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-socket\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.314145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.291283    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277" event={"ID":"12c229a2-f5f0-49d7-8812-6165d0efb27b","Type":"ContainerStarted","Data":"d24b7543a1d5b344019030b0d2e68bfcd6b95e6003679c4419bc9deaf615efec"}
Oct 30 21:57:07.424701 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.424645    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"]
Oct 30 21:57:07.466870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.466119    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh82j\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-kube-api-access-gh82j\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.467382 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.467321    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-envoy\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.476789 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.470996    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-istio-token\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.479050 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:57:07.479015    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7d7e86_f143_4876_a5a5_2175ec447a33.slice/crio-d6d16d94da6c0afdcbcdf6e527e059b6599d173cb939e7570809483b64fc5c1d WatchSource:0}: Error finding container d6d16d94da6c0afdcbcdf6e527e059b6599d173cb939e7570809483b64fc5c1d: Status 404 returned error can't find the container with id d6d16d94da6c0afdcbcdf6e527e059b6599d173cb939e7570809483b64fc5c1d
Oct 30 21:57:07.486106 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.486021    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f390701-2496-414e-ada8-90201df827ae-istio-podinfo\") pod \"e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") " pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.584615 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.584418    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec33fd1-e17b-4e07-8b4d-fe1d78a09152" path="/var/lib/kubelet/pods/eec33fd1-e17b-4e07-8b4d-fe1d78a09152/volumes"
Oct 30 21:57:07.667525 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.665880    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:07.977411 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.977375    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"]
Oct 30 21:57:07.981102 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.981057    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:57:07.981221 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.981200    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:57:07.981312 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:07.981294    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:57:08.337378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.336502    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" event={"ID":"8f390701-2496-414e-ada8-90201df827ae","Type":"ContainerStarted","Data":"eb407652e7ef23c3b7f08322a78a703d20c3f8f234ac63ab7a16b750d7c8a295"}
Oct 30 21:57:08.337378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.336560    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" event={"ID":"8f390701-2496-414e-ada8-90201df827ae","Type":"ContainerStarted","Data":"d497f860c5f474d834bf6002411fb2ea37aba809ae21844375a66d848f82ef49"}
Oct 30 21:57:08.342572 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.342534    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277" event={"ID":"12c229a2-f5f0-49d7-8812-6165d0efb27b","Type":"ContainerStarted","Data":"1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7"}
Oct 30 21:57:08.342750 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.342732    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:08.345289 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.345262    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq" event={"ID":"9a7d7e86-f143-4876-a5a5-2175ec447a33","Type":"ContainerStarted","Data":"dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f"}
Oct 30 21:57:08.345423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.345306    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq" event={"ID":"9a7d7e86-f143-4876-a5a5-2175ec447a33","Type":"ContainerStarted","Data":"d6d16d94da6c0afdcbcdf6e527e059b6599d173cb939e7570809483b64fc5c1d"}
Oct 30 21:57:08.345423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.345335    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:08.367309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.367268    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" podStartSLOduration=1.367256099 podStartE2EDuration="1.367256099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:08.364040689 +0000 UTC m=+1047.022136985" watchObservedRunningTime="2026-10-30 21:57:08.367256099 +0000 UTC m=+1047.025352322"
Oct 30 21:57:08.384103 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.384058    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq" podStartSLOduration=2.384041753 podStartE2EDuration="2.384041753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:08.380576501 +0000 UTC m=+1047.038672725" watchObservedRunningTime="2026-10-30 21:57:08.384041753 +0000 UTC m=+1047.042137988"
Oct 30 21:57:08.398655 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.398602    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277" podStartSLOduration=2.3985888859999998 podStartE2EDuration="2.398588886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:08.396345813 +0000 UTC m=+1047.054442036" watchObservedRunningTime="2026-10-30 21:57:08.398588886 +0000 UTC m=+1047.056685109"
Oct 30 21:57:08.666858 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.666830    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:08.767008 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.766975    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"]
Oct 30 21:57:08.810249 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:08.810216    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"]
Oct 30 21:57:09.727302 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:09.727260    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:10.365035 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:10.365006    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:10.366634 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:10.366613    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:57:11.138712 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.138681    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"]
Oct 30 21:57:11.143499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.143461    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.149113 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.149093    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ee0c56-serving-cert\""
Oct 30 21:57:11.162074 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.162048    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"]
Oct 30 21:57:11.227710 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.227677    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-metrics-certs\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.227861 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.227736    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.227912 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.227880    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2dh\" (UniqueName: \"kubernetes.io/projected/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-kube-api-access-4v2dh\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.227977 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.227938    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-maas-api-tls\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.339541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.336699    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-metrics-certs\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.339541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.336815    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.339541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.336992    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2dh\" (UniqueName: \"kubernetes.io/projected/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-kube-api-access-4v2dh\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.339541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.337101    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-maas-api-tls\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.339541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.338279    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-openshift-service-ca\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.359490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.356122    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-maas-api-tls\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.382106 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.372693    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-metrics-certs\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.382106 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.376332    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2dh\" (UniqueName: \"kubernetes.io/projected/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-kube-api-access-4v2dh\") pod \"maas-api-e2e-api-a-ee0c56-75776447b8-zqswx\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.459077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.458966    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:11.867886 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:11.867824    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"]
Oct 30 21:57:11.924498 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:57:11.923336    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0df5f56_45ee_42c3_9590_c0fdd7d56cfa.slice/crio-a6805c58ed5e7916dd40bbe1ba3cb54955f9c19d3510af5665af8207a8274510 WatchSource:0}: Error finding container a6805c58ed5e7916dd40bbe1ba3cb54955f9c19d3510af5665af8207a8274510: Status 404 returned error can't find the container with id a6805c58ed5e7916dd40bbe1ba3cb54955f9c19d3510af5665af8207a8274510
Oct 30 21:57:12.402559 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:12.402505    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx" event={"ID":"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa","Type":"ContainerStarted","Data":"a6805c58ed5e7916dd40bbe1ba3cb54955f9c19d3510af5665af8207a8274510"}
Oct 30 21:57:13.412166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:13.412130    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx" event={"ID":"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa","Type":"ContainerStarted","Data":"60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522"}
Oct 30 21:57:13.412623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:13.412279    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:13.489647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:13.489601    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx" podStartSLOduration=1.996302754 podStartE2EDuration="2.489586916s" totalImagesPullingTime="493.284162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:13.48660815 +0000 UTC m=+1052.144704373" watchObservedRunningTime="2026-10-30 21:57:13.489586916 +0000 UTC m=+1052.147683139"
Oct 30 21:57:13.767919 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:13.767877    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v" podUID="2b857469-db5e-4810-95b0-fd29fe7bbefd" containerName="payload-processing" containerID="cri-o://faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62" gracePeriod=25
Oct 30 21:57:13.811270 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:13.811235    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb" podUID="30f4f706-a1a4-47c6-b68a-73d76a146583" containerName="payload-pre-processing" containerID="cri-o://74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140" gracePeriod=25
Oct 30 21:57:14.202992 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.202922    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:57:14.219716 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.219616    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:57:14.303959 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.303931    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b857469-db5e-4810-95b0-fd29fe7bbefd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b857469-db5e-4810-95b0-fd29fe7bbefd-plugins-config-volume\") pod \"2b857469-db5e-4810-95b0-fd29fe7bbefd\" (UID: \"2b857469-db5e-4810-95b0-fd29fe7bbefd\") "
Oct 30 21:57:14.304131 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.304016    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b857469-db5e-4810-95b0-fd29fe7bbefd-kube-api-access-z4x4w\" (UniqueName: \"kubernetes.io/projected/2b857469-db5e-4810-95b0-fd29fe7bbefd-kube-api-access-z4x4w\") pod \"2b857469-db5e-4810-95b0-fd29fe7bbefd\" (UID: \"2b857469-db5e-4810-95b0-fd29fe7bbefd\") "
Oct 30 21:57:14.304379 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.304352    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b857469-db5e-4810-95b0-fd29fe7bbefd-plugins-config-volume" pod "2b857469-db5e-4810-95b0-fd29fe7bbefd" (UID: "2b857469-db5e-4810-95b0-fd29fe7bbefd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:14.304810 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.304790    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b857469-db5e-4810-95b0-fd29fe7bbefd-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:14.306445 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.306419    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b857469-db5e-4810-95b0-fd29fe7bbefd-kube-api-access-z4x4w" pod "2b857469-db5e-4810-95b0-fd29fe7bbefd" (UID: "2b857469-db5e-4810-95b0-fd29fe7bbefd"). InnerVolumeSpecName "kube-api-access-z4x4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:14.406506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.406415    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f4f706-a1a4-47c6-b68a-73d76a146583-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f4f706-a1a4-47c6-b68a-73d76a146583-plugins-config-volume\") pod \"30f4f706-a1a4-47c6-b68a-73d76a146583\" (UID: \"30f4f706-a1a4-47c6-b68a-73d76a146583\") "
Oct 30 21:57:14.406657 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.406508    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f4f706-a1a4-47c6-b68a-73d76a146583-kube-api-access-l2g4p\" (UniqueName: \"kubernetes.io/projected/30f4f706-a1a4-47c6-b68a-73d76a146583-kube-api-access-l2g4p\") pod \"30f4f706-a1a4-47c6-b68a-73d76a146583\" (UID: \"30f4f706-a1a4-47c6-b68a-73d76a146583\") "
Oct 30 21:57:14.406811 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.406780    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f4f706-a1a4-47c6-b68a-73d76a146583-plugins-config-volume" pod "30f4f706-a1a4-47c6-b68a-73d76a146583" (UID: "30f4f706-a1a4-47c6-b68a-73d76a146583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:14.406942 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.406928    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30f4f706-a1a4-47c6-b68a-73d76a146583-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:14.406991 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.406945    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4x4w\" (UniqueName: \"kubernetes.io/projected/2b857469-db5e-4810-95b0-fd29fe7bbefd-kube-api-access-z4x4w\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:14.408688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.408670    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f4f706-a1a4-47c6-b68a-73d76a146583-kube-api-access-l2g4p" pod "30f4f706-a1a4-47c6-b68a-73d76a146583" (UID: "30f4f706-a1a4-47c6-b68a-73d76a146583"). InnerVolumeSpecName "kube-api-access-l2g4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:14.419967 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.419943    4282 generic.go:356] "Generic (PLEG): container finished" podID="30f4f706-a1a4-47c6-b68a-73d76a146583" containerID="74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140" exitCode=0
Oct 30 21:57:14.420274 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.419984    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"
Oct 30 21:57:14.420274 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.420021    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb" event={"ID":"30f4f706-a1a4-47c6-b68a-73d76a146583","Type":"ContainerDied","Data":"74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140"}
Oct 30 21:57:14.420274 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.420071    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb" event={"ID":"30f4f706-a1a4-47c6-b68a-73d76a146583","Type":"ContainerDied","Data":"5ee1ee36e9e5ae3a0cc5ed760f6ebfe11c8e8f60feed4a8ac23029391eff6989"}
Oct 30 21:57:14.420274 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.420093    4282 scope.go:122] "RemoveContainer" containerID="74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140"
Oct 30 21:57:14.421718 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.421695    4282 generic.go:356] "Generic (PLEG): container finished" podID="2b857469-db5e-4810-95b0-fd29fe7bbefd" containerID="faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62" exitCode=0
Oct 30 21:57:14.421823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.421803    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v" event={"ID":"2b857469-db5e-4810-95b0-fd29fe7bbefd","Type":"ContainerDied","Data":"faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62"}
Oct 30 21:57:14.421882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.421864    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v" event={"ID":"2b857469-db5e-4810-95b0-fd29fe7bbefd","Type":"ContainerDied","Data":"85bc971522362cbc85681acd22b92daf847ad54c9425536f98c8c4e722e31f94"}
Oct 30 21:57:14.422006 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.421974    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"
Oct 30 21:57:14.431989 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.431969    4282 scope.go:122] "RemoveContainer" containerID="74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140"
Oct 30 21:57:14.432331 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:14.432252    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140\": container with ID starting with 74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140 not found: ID does not exist" containerID="74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140"
Oct 30 21:57:14.432331 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.432287    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140"} err="failed to get container status \"74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140\": rpc error: code = NotFound desc = could not find container \"74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140\": container with ID starting with 74e35a2b1ea9f932831a9485a2c43aa28abe0497da117214bac7090b68faa140 not found: ID does not exist"
Oct 30 21:57:14.432331 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.432308    4282 scope.go:122] "RemoveContainer" containerID="faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62"
Oct 30 21:57:14.441436 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.441418    4282 scope.go:122] "RemoveContainer" containerID="faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62"
Oct 30 21:57:14.441741 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:14.441724    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62\": container with ID starting with faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62 not found: ID does not exist" containerID="faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62"
Oct 30 21:57:14.441797 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.441748    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62"} err="failed to get container status \"faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62\": rpc error: code = NotFound desc = could not find container \"faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62\": container with ID starting with faefe1358cd26cd972ecc8f13decb51c1c65e4613b753b52b53a43fe45932c62 not found: ID does not exist"
Oct 30 21:57:14.457752 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.457730    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"]
Oct 30 21:57:14.463642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.463621    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-79724b-758f5dfc7f-xpq4v"]
Oct 30 21:57:14.479169 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.479150    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"]
Oct 30 21:57:14.489097 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.489078    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-79724b-6789755dc5-qzrcb"]
Oct 30 21:57:14.508881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:14.508859    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2g4p\" (UniqueName: \"kubernetes.io/projected/30f4f706-a1a4-47c6-b68a-73d76a146583-kube-api-access-l2g4p\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:15.464283 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:15.464251    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b857469-db5e-4810-95b0-fd29fe7bbefd" path="/var/lib/kubelet/pods/2b857469-db5e-4810-95b0-fd29fe7bbefd/volumes"
Oct 30 21:57:15.464750 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:15.464714    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f4f706-a1a4-47c6-b68a-73d76a146583" path="/var/lib/kubelet/pods/30f4f706-a1a4-47c6-b68a-73d76a146583/volumes"
Oct 30 21:57:16.969423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:16.969381    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:57:18.104599 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.104566    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"]
Oct 30 21:57:18.106436 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.106406    4282 cpu_manager.go:414] "containerMap: removing container" podUID="30f4f706-a1a4-47c6-b68a-73d76a146583" containerName="payload-pre-processing"
Oct 30 21:57:18.106436 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.106435    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f4f706-a1a4-47c6-b68a-73d76a146583" containerName="payload-pre-processing"
Oct 30 21:57:18.106667 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.106501    4282 cpu_manager.go:414] "containerMap: removing container" podUID="2b857469-db5e-4810-95b0-fd29fe7bbefd" containerName="payload-processing"
Oct 30 21:57:18.106667 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.106510    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b857469-db5e-4810-95b0-fd29fe7bbefd" containerName="payload-processing"
Oct 30 21:57:18.106804 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.106694    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b857469-db5e-4810-95b0-fd29fe7bbefd" containerName="payload-processing"
Oct 30 21:57:18.106804 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.106716    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f4f706-a1a4-47c6-b68a-73d76a146583" containerName="payload-pre-processing"
Oct 30 21:57:18.112301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.112279    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.123965 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.123942    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-56c0af-serving-cert\""
Oct 30 21:57:18.128321 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.128301    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"]
Oct 30 21:57:18.286711 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.286658    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-maas-api-tls\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.287429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.287399    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-openshift-service-ca\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.287804 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.287783    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hz9c\" (UniqueName: \"kubernetes.io/projected/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-kube-api-access-8hz9c\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.295488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.287955    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-metrics-certs\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.404540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.401571    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hz9c\" (UniqueName: \"kubernetes.io/projected/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-kube-api-access-8hz9c\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.404540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.401649    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-metrics-certs\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.404540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.401831    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-maas-api-tls\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.404540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.401911    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-openshift-service-ca\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.404540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.402812    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-openshift-service-ca\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.423120 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.423081    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-metrics-certs\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.425254 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.425216    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hz9c\" (UniqueName: \"kubernetes.io/projected/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-kube-api-access-8hz9c\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.460425 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.460388    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-maas-api-tls\") pod \"maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:18.746494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:18.734821    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:19.395565 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:19.395525    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:57:19.397665 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:19.397645    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:57:19.525747 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:19.525668    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:57:19.573450 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:19.573412    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"]
Oct 30 21:57:20.395222 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:20.395191    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:57:20.395526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:20.395497    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:20.780128 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:20.780091    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd" event={"ID":"1c6cc795-b6a1-4a4a-80d4-02645ee38f27","Type":"ContainerStarted","Data":"4a27ca9e929c76e4abbe6b3e9638927a927ea3e1c24731a01d92f1da00529403"}
Oct 30 21:57:20.896070 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:20.864677    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7694-498663"]
Oct 30 21:57:20.896070 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:20.872033    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:20.896070 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:20.895912    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sfm7d\""
Oct 30 21:57:21.029539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.028656    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-498663"]
Oct 30 21:57:21.074051 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.046751    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psh5h\" (UniqueName: \"kubernetes.io/projected/91cb9d6e-2399-4d8e-a48c-179b2cd2291d-kube-api-access-psh5h\") pod \"test-curl-7694-498663\" (UID: \"91cb9d6e-2399-4d8e-a48c-179b2cd2291d\") " pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:21.074328 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.074278    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:57:21.149167 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.149134    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psh5h\" (UniqueName: \"kubernetes.io/projected/91cb9d6e-2399-4d8e-a48c-179b2cd2291d-kube-api-access-psh5h\") pod \"test-curl-7694-498663\" (UID: \"91cb9d6e-2399-4d8e-a48c-179b2cd2291d\") " pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:21.287984 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.287760    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psh5h\" (UniqueName: \"kubernetes.io/projected/91cb9d6e-2399-4d8e-a48c-179b2cd2291d-kube-api-access-psh5h\") pod \"test-curl-7694-498663\" (UID: \"91cb9d6e-2399-4d8e-a48c-179b2cd2291d\") " pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:21.309589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.309538    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:57:21.390929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.390900    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:21.788145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.788109    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd" event={"ID":"1c6cc795-b6a1-4a4a-80d4-02645ee38f27","Type":"ContainerStarted","Data":"9365779ffdc186a1fb3067e7f16af7e4d377ea836d63b51a37dce176004f64ef"}
Oct 30 21:57:21.788555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.788319    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:21.810375 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.810331    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd" podStartSLOduration=2.546156255 podStartE2EDuration="3.810317032s" totalImagesPullingTime="1.264160777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:21.806558313 +0000 UTC m=+1060.464654536" watchObservedRunningTime="2026-10-30 21:57:21.810317032 +0000 UTC m=+1060.468413255"
Oct 30 21:57:21.847996 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:21.847971    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-498663"]
Oct 30 21:57:21.848581 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:57:21.848552    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91cb9d6e_2399_4d8e_a48c_179b2cd2291d.slice/crio-d7668e9c2ed196ea61a88b2c351735ad790108f56ce6494a9cafdb87d4a61d22 WatchSource:0}: Error finding container d7668e9c2ed196ea61a88b2c351735ad790108f56ce6494a9cafdb87d4a61d22: Status 404 returned error can't find the container with id d7668e9c2ed196ea61a88b2c351735ad790108f56ce6494a9cafdb87d4a61d22
Oct 30 21:57:22.797066 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:22.797034    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-498663" event={"ID":"91cb9d6e-2399-4d8e-a48c-179b2cd2291d","Type":"ContainerStarted","Data":"d7668e9c2ed196ea61a88b2c351735ad790108f56ce6494a9cafdb87d4a61d22"}
Oct 30 21:57:24.331756 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.331725    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"]
Oct 30 21:57:24.332281 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.331908    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x" podUID="3895fd1a-8e26-467b-aee3-f8b5e271993f" containerName="maas-api" containerID="cri-o://6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc" gracePeriod=30
Oct 30 21:57:24.707024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.706999    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:57:24.957703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.956719    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-metrics-certs\") pod \"3895fd1a-8e26-467b-aee3-f8b5e271993f\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") "
Oct 30 21:57:24.957703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.956771    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3895fd1a-8e26-467b-aee3-f8b5e271993f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895fd1a-8e26-467b-aee3-f8b5e271993f-openshift-service-ca\") pod \"3895fd1a-8e26-467b-aee3-f8b5e271993f\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") "
Oct 30 21:57:24.957703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.957060    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3895fd1a-8e26-467b-aee3-f8b5e271993f-kube-api-access-mc24d\" (UniqueName: \"kubernetes.io/projected/3895fd1a-8e26-467b-aee3-f8b5e271993f-kube-api-access-mc24d\") pod \"3895fd1a-8e26-467b-aee3-f8b5e271993f\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") "
Oct 30 21:57:24.957703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.957153    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-maas-api-tls\") pod \"3895fd1a-8e26-467b-aee3-f8b5e271993f\" (UID: \"3895fd1a-8e26-467b-aee3-f8b5e271993f\") "
Oct 30 21:57:24.971493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.963803    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3895fd1a-8e26-467b-aee3-f8b5e271993f-openshift-service-ca" pod "3895fd1a-8e26-467b-aee3-f8b5e271993f" (UID: "3895fd1a-8e26-467b-aee3-f8b5e271993f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:24.971493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.970927    4282 generic.go:356] "Generic (PLEG): container finished" podID="3895fd1a-8e26-467b-aee3-f8b5e271993f" containerID="6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc" exitCode=0
Oct 30 21:57:24.971493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.970999    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x" event={"ID":"3895fd1a-8e26-467b-aee3-f8b5e271993f","Type":"ContainerDied","Data":"6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc"}
Oct 30 21:57:24.971493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.971061    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x" event={"ID":"3895fd1a-8e26-467b-aee3-f8b5e271993f","Type":"ContainerDied","Data":"b5649276c75d404c48f8733f44b9a5233c5795b03186fe3613c9d733e7cdfc7d"}
Oct 30 21:57:24.971493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.971136    4282 scope.go:122] "RemoveContainer" containerID="6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc"
Oct 30 21:57:24.971493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.971304    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"
Oct 30 21:57:24.994765 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:24.994188    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-maas-api-tls" pod "3895fd1a-8e26-467b-aee3-f8b5e271993f" (UID: "3895fd1a-8e26-467b-aee3-f8b5e271993f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:25.002323 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.002257    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3895fd1a-8e26-467b-aee3-f8b5e271993f-kube-api-access-mc24d" pod "3895fd1a-8e26-467b-aee3-f8b5e271993f" (UID: "3895fd1a-8e26-467b-aee3-f8b5e271993f"). InnerVolumeSpecName "kube-api-access-mc24d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:25.002728 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.002689    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-metrics-certs" pod "3895fd1a-8e26-467b-aee3-f8b5e271993f" (UID: "3895fd1a-8e26-467b-aee3-f8b5e271993f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:25.020751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.020624    4282 scope.go:122] "RemoveContainer" containerID="6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc"
Oct 30 21:57:25.021040 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:25.020994    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc\": container with ID starting with 6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc not found: ID does not exist" containerID="6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc"
Oct 30 21:57:25.021135 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.021031    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc"} err="failed to get container status \"6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc\": rpc error: code = NotFound desc = could not find container \"6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc\": container with ID starting with 6d824fd8e1e1b33fc48ce14db48718b8766f334ebce07b685cbefe96fefc63bc not found: ID does not exist"
Oct 30 21:57:25.074751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.073002    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc24d\" (UniqueName: \"kubernetes.io/projected/3895fd1a-8e26-467b-aee3-f8b5e271993f-kube-api-access-mc24d\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:25.074751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.073041    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:25.074751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.073056    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3895fd1a-8e26-467b-aee3-f8b5e271993f-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:25.074751 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.073070    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3895fd1a-8e26-467b-aee3-f8b5e271993f-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:25.312806 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.312774    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"]
Oct 30 21:57:25.328309 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.328281    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-79724b-94dd9b8d8-7sg8x"]
Oct 30 21:57:25.464747 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.464714    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3895fd1a-8e26-467b-aee3-f8b5e271993f" path="/var/lib/kubelet/pods/3895fd1a-8e26-467b-aee3-f8b5e271993f/volumes"
Oct 30 21:57:25.980129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.980096    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-498663" event={"ID":"91cb9d6e-2399-4d8e-a48c-179b2cd2291d","Type":"ContainerStarted","Data":"94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f"}
Oct 30 21:57:25.998206 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:25.998157    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7694-498663" podStartSLOduration=3.193039237 podStartE2EDuration="5.998142967s" totalImagesPullingTime="2.80510373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:25.995854717 +0000 UTC m=+1064.653950939" watchObservedRunningTime="2026-10-30 21:57:25.998142967 +0000 UTC m=+1064.656239189"
Oct 30 21:57:26.939626 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:26.939551    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7694-498663"]
Oct 30 21:57:26.940166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:26.939643    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7694-498663"]
Oct 30 21:57:27.802576 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:27.802549    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:57:28.002612 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:28.002575    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7694-498663" podUID="91cb9d6e-2399-4d8e-a48c-179b2cd2291d" containerName="test-curl-7694-498663" containerID="cri-o://94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f" gracePeriod=2
Oct 30 21:57:30.220195 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:30.220172    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:30.330529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:30.330452    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91cb9d6e-2399-4d8e-a48c-179b2cd2291d-kube-api-access-psh5h\" (UniqueName: \"kubernetes.io/projected/91cb9d6e-2399-4d8e-a48c-179b2cd2291d-kube-api-access-psh5h\") pod \"91cb9d6e-2399-4d8e-a48c-179b2cd2291d\" (UID: \"91cb9d6e-2399-4d8e-a48c-179b2cd2291d\") "
Oct 30 21:57:30.332822 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:30.332800    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91cb9d6e-2399-4d8e-a48c-179b2cd2291d-kube-api-access-psh5h" pod "91cb9d6e-2399-4d8e-a48c-179b2cd2291d" (UID: "91cb9d6e-2399-4d8e-a48c-179b2cd2291d"). InnerVolumeSpecName "kube-api-access-psh5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:30.433186 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:30.433154    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psh5h\" (UniqueName: \"kubernetes.io/projected/91cb9d6e-2399-4d8e-a48c-179b2cd2291d-kube-api-access-psh5h\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:31.034640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.034605    4282 generic.go:356] "Generic (PLEG): container finished" podID="91cb9d6e-2399-4d8e-a48c-179b2cd2291d" containerID="94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f" exitCode=137
Oct 30 21:57:31.034818 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.034658    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:31.034818 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.034676    4282 scope.go:122] "RemoveContainer" containerID="94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f"
Oct 30 21:57:31.044660 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:31.044629    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7694-498663\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="91cb9d6e-2399-4d8e-a48c-179b2cd2291d" pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:31.045210 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.045195    4282 scope.go:122] "RemoveContainer" containerID="94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f"
Oct 30 21:57:31.045530 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:31.045460    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f\": container with ID starting with 94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f not found: ID does not exist" containerID="94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f"
Oct 30 21:57:31.045630 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.045537    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f"} err="failed to get container status \"94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f\": rpc error: code = NotFound desc = could not find container \"94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f\": container with ID starting with 94fd63f155484130b91a275ed6dbeed5f27f84fb0e2d2d2778e7648f42bff23f not found: ID does not exist"
Oct 30 21:57:31.462633 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:31.462596    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7694-498663\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="91cb9d6e-2399-4d8e-a48c-179b2cd2291d" pod="opendatahub/test-curl-7694-498663"
Oct 30 21:57:31.465018 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.464994    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91cb9d6e-2399-4d8e-a48c-179b2cd2291d" path="/var/lib/kubelet/pods/91cb9d6e-2399-4d8e-a48c-179b2cd2291d/volumes"
Oct 30 21:57:31.790628 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.790554    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"]
Oct 30 21:57:31.791163 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.791149    4282 cpu_manager.go:414] "containerMap: removing container" podUID="91cb9d6e-2399-4d8e-a48c-179b2cd2291d" containerName="test-curl-7694-498663"
Oct 30 21:57:31.791208 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.791164    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91cb9d6e-2399-4d8e-a48c-179b2cd2291d" containerName="test-curl-7694-498663"
Oct 30 21:57:31.791244 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.791211    4282 cpu_manager.go:414] "containerMap: removing container" podUID="3895fd1a-8e26-467b-aee3-f8b5e271993f" containerName="maas-api"
Oct 30 21:57:31.791244 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.791216    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3895fd1a-8e26-467b-aee3-f8b5e271993f" containerName="maas-api"
Oct 30 21:57:31.791325 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.791299    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3895fd1a-8e26-467b-aee3-f8b5e271993f" containerName="maas-api"
Oct 30 21:57:31.791325 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.791309    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="91cb9d6e-2399-4d8e-a48c-179b2cd2291d" containerName="test-curl-7694-498663"
Oct 30 21:57:31.796397 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.796379    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:31.810460 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.810439    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"]
Oct 30 21:57:31.811203 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.811176    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4bf337\""
Oct 30 21:57:31.819697 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.819671    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"]
Oct 30 21:57:31.824219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.824199    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:31.838209 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.838187    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"]
Oct 30 21:57:31.854838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.854805    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4bf337-dockercfg-g4vkv\""
Oct 30 21:57:31.893675 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.893648    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/991c83df-52d4-4454-b757-27129f8f41c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4bf337-658d8d687-trgsf\" (UID: \"991c83df-52d4-4454-b757-27129f8f41c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:31.893838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.893725    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5gj\" (UniqueName: \"kubernetes.io/projected/991c83df-52d4-4454-b757-27129f8f41c5-kube-api-access-hr5gj\") pod \"payload-processing-e2e-api-b-4bf337-658d8d687-trgsf\" (UID: \"991c83df-52d4-4454-b757-27129f8f41c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:31.893838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.893823    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149abdb-cbb4-45d6-88d3-008adf585246-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p\" (UID: \"7149abdb-cbb4-45d6-88d3-008adf585246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:31.893929 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.893919    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqvrk\" (UniqueName: \"kubernetes.io/projected/7149abdb-cbb4-45d6-88d3-008adf585246-kube-api-access-zqvrk\") pod \"payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p\" (UID: \"7149abdb-cbb4-45d6-88d3-008adf585246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:32.003502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.997874    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5gj\" (UniqueName: \"kubernetes.io/projected/991c83df-52d4-4454-b757-27129f8f41c5-kube-api-access-hr5gj\") pod \"payload-processing-e2e-api-b-4bf337-658d8d687-trgsf\" (UID: \"991c83df-52d4-4454-b757-27129f8f41c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:32.003502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.997997    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149abdb-cbb4-45d6-88d3-008adf585246-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p\" (UID: \"7149abdb-cbb4-45d6-88d3-008adf585246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:32.003502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.998097    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvrk\" (UniqueName: \"kubernetes.io/projected/7149abdb-cbb4-45d6-88d3-008adf585246-kube-api-access-zqvrk\") pod \"payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p\" (UID: \"7149abdb-cbb4-45d6-88d3-008adf585246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:32.003502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.998286    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/991c83df-52d4-4454-b757-27129f8f41c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4bf337-658d8d687-trgsf\" (UID: \"991c83df-52d4-4454-b757-27129f8f41c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:32.003502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.999299    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/991c83df-52d4-4454-b757-27129f8f41c5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4bf337-658d8d687-trgsf\" (UID: \"991c83df-52d4-4454-b757-27129f8f41c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:32.003502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:31.999452    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149abdb-cbb4-45d6-88d3-008adf585246-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p\" (UID: \"7149abdb-cbb4-45d6-88d3-008adf585246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:32.036103 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:32.034109    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqvrk\" (UniqueName: \"kubernetes.io/projected/7149abdb-cbb4-45d6-88d3-008adf585246-kube-api-access-zqvrk\") pod \"payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p\" (UID: \"7149abdb-cbb4-45d6-88d3-008adf585246\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:32.061817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:32.061741    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5gj\" (UniqueName: \"kubernetes.io/projected/991c83df-52d4-4454-b757-27129f8f41c5-kube-api-access-hr5gj\") pod \"payload-processing-e2e-api-b-4bf337-658d8d687-trgsf\" (UID: \"991c83df-52d4-4454-b757-27129f8f41c5\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:32.116938 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:32.116675    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:32.145868 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:32.145834    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:32.627162 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:32.627135    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"]
Oct 30 21:57:32.627539 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:57:32.627372    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991c83df_52d4_4454_b757_27129f8f41c5.slice/crio-3551ffe9b617b95f99711ef146611332d36bbb1d32d65038ceb7193e0d22f480 WatchSource:0}: Error finding container 3551ffe9b617b95f99711ef146611332d36bbb1d32d65038ceb7193e0d22f480: Status 404 returned error can't find the container with id 3551ffe9b617b95f99711ef146611332d36bbb1d32d65038ceb7193e0d22f480
Oct 30 21:57:32.726574 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:32.726539    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"]
Oct 30 21:57:33.250193 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:33.250159    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p" event={"ID":"7149abdb-cbb4-45d6-88d3-008adf585246","Type":"ContainerStarted","Data":"2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48"}
Oct 30 21:57:33.250362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:33.250212    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p" event={"ID":"7149abdb-cbb4-45d6-88d3-008adf585246","Type":"ContainerStarted","Data":"e269a8b504bc107f3010cd1274c2e90b5989ad4a556cca0dcd365bd29c4a6b4b"}
Oct 30 21:57:33.250362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:33.250251    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:33.251943 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:33.251926    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf" event={"ID":"991c83df-52d4-4454-b757-27129f8f41c5","Type":"ContainerStarted","Data":"2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc"}
Oct 30 21:57:33.252019 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:33.251959    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf" event={"ID":"991c83df-52d4-4454-b757-27129f8f41c5","Type":"ContainerStarted","Data":"3551ffe9b617b95f99711ef146611332d36bbb1d32d65038ceb7193e0d22f480"}
Oct 30 21:57:33.252019 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:33.251988    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:33.267951 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:33.267910    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p" podStartSLOduration=2.267898614 podStartE2EDuration="2.267898614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:33.26675161 +0000 UTC m=+1071.924847844" watchObservedRunningTime="2026-10-30 21:57:33.267898614 +0000 UTC m=+1071.925994836"
Oct 30 21:57:33.284936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:33.284897    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf" podStartSLOduration=2.284881717 podStartE2EDuration="2.284881717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:33.283300202 +0000 UTC m=+1071.941396426" watchObservedRunningTime="2026-10-30 21:57:33.284881717 +0000 UTC m=+1071.942977984"
Oct 30 21:57:34.648881 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:34.648844    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:57:37.329985 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:37.329953    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"]
Oct 30 21:57:38.136109 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.136079    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"]
Oct 30 21:57:38.142063 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.142040    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.147952 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.147929    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4bf337-serving-cert\""
Oct 30 21:57:38.154387 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.154359    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"]
Oct 30 21:57:38.190727 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.190692    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114c471-8f92-4e57-a34a-deda0832dd12-openshift-service-ca\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.190933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.190904    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-maas-api-tls\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.191081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.191048    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-metrics-certs\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.191232 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.191194    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p26t5\" (UniqueName: \"kubernetes.io/projected/7114c471-8f92-4e57-a34a-deda0832dd12-kube-api-access-p26t5\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.292438 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.292408    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114c471-8f92-4e57-a34a-deda0832dd12-openshift-service-ca\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.292630 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.292583    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-maas-api-tls\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.292688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.292658    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-metrics-certs\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.292782 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.292764    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p26t5\" (UniqueName: \"kubernetes.io/projected/7114c471-8f92-4e57-a34a-deda0832dd12-kube-api-access-p26t5\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.293170 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.293136    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114c471-8f92-4e57-a34a-deda0832dd12-openshift-service-ca\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.295216 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.295197    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-metrics-certs\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.295303 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.295242    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-maas-api-tls\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.355506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.353174    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p26t5\" (UniqueName: \"kubernetes.io/projected/7114c471-8f92-4e57-a34a-deda0832dd12-kube-api-access-p26t5\") pod \"maas-api-e2e-api-b-4bf337-b996b7984-98lrp\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.467501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.467435    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:38.849100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:38.845934    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"]
Oct 30 21:57:39.321381 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:39.321349    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp" event={"ID":"7114c471-8f92-4e57-a34a-deda0832dd12","Type":"ContainerStarted","Data":"8bc4cf5a4f8e9feb28da6b4adf867cfa117c6cd0c5295e947a7e697597a12ade"}
Oct 30 21:57:40.331541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:40.331508    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp" event={"ID":"7114c471-8f92-4e57-a34a-deda0832dd12","Type":"ContainerStarted","Data":"cbc01f4c5f2fe1d00bb6e9147b790e0da365c91f0248acb0899db4b85e527452"}
Oct 30 21:57:40.331927 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:40.331621    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:40.352714 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:40.352670    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp" podStartSLOduration=1.913363694 podStartE2EDuration="2.352656854s" totalImagesPullingTime="439.29316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:40.349983217 +0000 UTC m=+1079.008079439" watchObservedRunningTime="2026-10-30 21:57:40.352656854 +0000 UTC m=+1079.010753076"
Oct 30 21:57:42.331149 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:42.331111    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc" podUID="c42d6a09-5bf9-4e10-8576-44ad7442e7f0" containerName="payload-pre-processing" containerID="cri-o://c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed" gracePeriod=25
Oct 30 21:57:42.615623 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:42.615595    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:57:42.630134 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:42.630110    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-kube-api-access-pzg9q\" (UniqueName: \"kubernetes.io/projected/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-kube-api-access-pzg9q\") pod \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\" (UID: \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\") "
Oct 30 21:57:42.630268 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:42.630257    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-plugins-config-volume\") pod \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\" (UID: \"c42d6a09-5bf9-4e10-8576-44ad7442e7f0\") "
Oct 30 21:57:42.630652 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:42.630624    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-plugins-config-volume" pod "c42d6a09-5bf9-4e10-8576-44ad7442e7f0" (UID: "c42d6a09-5bf9-4e10-8576-44ad7442e7f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:42.630914 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:42.630898    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:42.632227 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:42.632208    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-kube-api-access-pzg9q" pod "c42d6a09-5bf9-4e10-8576-44ad7442e7f0" (UID: "c42d6a09-5bf9-4e10-8576-44ad7442e7f0"). InnerVolumeSpecName "kube-api-access-pzg9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:42.732262 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:42.732228    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzg9q\" (UniqueName: \"kubernetes.io/projected/c42d6a09-5bf9-4e10-8576-44ad7442e7f0-kube-api-access-pzg9q\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:43.141179 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.141150    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"]
Oct 30 21:57:43.155171 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.155145    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"]
Oct 30 21:57:43.364704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.364621    4282 generic.go:356] "Generic (PLEG): container finished" podID="c42d6a09-5bf9-4e10-8576-44ad7442e7f0" containerID="c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed" exitCode=0
Oct 30 21:57:43.364704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.364679    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"
Oct 30 21:57:43.364704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.364688    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc" event={"ID":"c42d6a09-5bf9-4e10-8576-44ad7442e7f0","Type":"ContainerDied","Data":"c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed"}
Oct 30 21:57:43.365440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.364757    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc" event={"ID":"c42d6a09-5bf9-4e10-8576-44ad7442e7f0","Type":"ContainerDied","Data":"d10baeb8658c98b443062baa4021d770c7f4d6255deb721c5b7c0c17558e59f7"}
Oct 30 21:57:43.365440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.364781    4282 scope.go:122] "RemoveContainer" containerID="c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed"
Oct 30 21:57:43.375804 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.375778    4282 scope.go:122] "RemoveContainer" containerID="c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed"
Oct 30 21:57:43.376071 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:57:43.376051    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed\": container with ID starting with c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed not found: ID does not exist" containerID="c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed"
Oct 30 21:57:43.376134 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.376084    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed"} err="failed to get container status \"c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed\": rpc error: code = NotFound desc = could not find container \"c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed\": container with ID starting with c57faa1fc8ac2a272264bb90eb327c71cb25376758b00759f5db414a2f6182ed not found: ID does not exist"
Oct 30 21:57:43.390387 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.390363    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"]
Oct 30 21:57:43.395158 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.395136    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-689a2563-677cbdc69f-vkjgc"]
Oct 30 21:57:43.475504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:43.475005    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c42d6a09-5bf9-4e10-8576-44ad7442e7f0" path="/var/lib/kubelet/pods/c42d6a09-5bf9-4e10-8576-44ad7442e7f0/volumes"
Oct 30 21:57:44.261447 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:44.261420    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:57:44.261625 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:44.261506    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:57:46.357778 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:46.357751    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:57:48.142504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.142433    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2" podUID="0d0015e3-f2b6-4d37-9f3c-b183498228fd" containerName="payload-processing" containerID="cri-o://ee9f1bee0a58a4a254b67b12989e177c148ffa486bd068fce1ff38757ccc5517" gracePeriod=25
Oct 30 21:57:48.156995 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.156964    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285" podUID="08a29d12-2b5f-43a4-a00b-07e87430976b" containerName="payload-pre-processing" containerID="cri-o://4e6d275495bf48c6d95f3410cabb154711da2dce9ed491f7247c6899a3b4a624" gracePeriod=25
Oct 30 21:57:48.409762 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.409563    4282 generic.go:356] "Generic (PLEG): container finished" podID="08a29d12-2b5f-43a4-a00b-07e87430976b" containerID="4e6d275495bf48c6d95f3410cabb154711da2dce9ed491f7247c6899a3b4a624" exitCode=0
Oct 30 21:57:48.409762 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.409662    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285" event={"ID":"08a29d12-2b5f-43a4-a00b-07e87430976b","Type":"ContainerDied","Data":"4e6d275495bf48c6d95f3410cabb154711da2dce9ed491f7247c6899a3b4a624"}
Oct 30 21:57:48.412234 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.412180    4282 generic.go:356] "Generic (PLEG): container finished" podID="0d0015e3-f2b6-4d37-9f3c-b183498228fd" containerID="ee9f1bee0a58a4a254b67b12989e177c148ffa486bd068fce1ff38757ccc5517" exitCode=0
Oct 30 21:57:48.412399 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.412379    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2" event={"ID":"0d0015e3-f2b6-4d37-9f3c-b183498228fd","Type":"ContainerDied","Data":"ee9f1bee0a58a4a254b67b12989e177c148ffa486bd068fce1ff38757ccc5517"}
Oct 30 21:57:48.438188 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.438162    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:57:48.535387 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.535365    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:57:48.543261 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.543238    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a29d12-2b5f-43a4-a00b-07e87430976b-kube-api-access-7qmc5\" (UniqueName: \"kubernetes.io/projected/08a29d12-2b5f-43a4-a00b-07e87430976b-kube-api-access-7qmc5\") pod \"08a29d12-2b5f-43a4-a00b-07e87430976b\" (UID: \"08a29d12-2b5f-43a4-a00b-07e87430976b\") "
Oct 30 21:57:48.543368 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.543273    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0015e3-f2b6-4d37-9f3c-b183498228fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0015e3-f2b6-4d37-9f3c-b183498228fd-plugins-config-volume\") pod \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\" (UID: \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\") "
Oct 30 21:57:48.543368 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.543292    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a29d12-2b5f-43a4-a00b-07e87430976b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a29d12-2b5f-43a4-a00b-07e87430976b-plugins-config-volume\") pod \"08a29d12-2b5f-43a4-a00b-07e87430976b\" (UID: \"08a29d12-2b5f-43a4-a00b-07e87430976b\") "
Oct 30 21:57:48.543368 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.543324    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0015e3-f2b6-4d37-9f3c-b183498228fd-kube-api-access-fgc5p\" (UniqueName: \"kubernetes.io/projected/0d0015e3-f2b6-4d37-9f3c-b183498228fd-kube-api-access-fgc5p\") pod \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\" (UID: \"0d0015e3-f2b6-4d37-9f3c-b183498228fd\") "
Oct 30 21:57:48.543672 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.543650    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0015e3-f2b6-4d37-9f3c-b183498228fd-plugins-config-volume" pod "0d0015e3-f2b6-4d37-9f3c-b183498228fd" (UID: "0d0015e3-f2b6-4d37-9f3c-b183498228fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:48.543734 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.543665    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a29d12-2b5f-43a4-a00b-07e87430976b-plugins-config-volume" pod "08a29d12-2b5f-43a4-a00b-07e87430976b" (UID: "08a29d12-2b5f-43a4-a00b-07e87430976b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:48.543968 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.543949    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0015e3-f2b6-4d37-9f3c-b183498228fd-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:48.544024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.543974    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08a29d12-2b5f-43a4-a00b-07e87430976b-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:48.545547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.545528    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0015e3-f2b6-4d37-9f3c-b183498228fd-kube-api-access-fgc5p" pod "0d0015e3-f2b6-4d37-9f3c-b183498228fd" (UID: "0d0015e3-f2b6-4d37-9f3c-b183498228fd"). InnerVolumeSpecName "kube-api-access-fgc5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:48.545613 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.545543    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a29d12-2b5f-43a4-a00b-07e87430976b-kube-api-access-7qmc5" pod "08a29d12-2b5f-43a4-a00b-07e87430976b" (UID: "08a29d12-2b5f-43a4-a00b-07e87430976b"). InnerVolumeSpecName "kube-api-access-7qmc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:48.663542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.660690    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qmc5\" (UniqueName: \"kubernetes.io/projected/08a29d12-2b5f-43a4-a00b-07e87430976b-kube-api-access-7qmc5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:48.663542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:48.660735    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgc5p\" (UniqueName: \"kubernetes.io/projected/0d0015e3-f2b6-4d37-9f3c-b183498228fd-kube-api-access-fgc5p\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:49.421881 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.421857    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"
Oct 30 21:57:49.422314 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.421860    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285" event={"ID":"08a29d12-2b5f-43a4-a00b-07e87430976b","Type":"ContainerDied","Data":"9c6112c11302ef72e1609abd5b115614b2e512bed096c772a418a7e6d2d22c93"}
Oct 30 21:57:49.422314 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.422011    4282 scope.go:122] "RemoveContainer" containerID="4e6d275495bf48c6d95f3410cabb154711da2dce9ed491f7247c6899a3b4a624"
Oct 30 21:57:49.423708 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.423671    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"
Oct 30 21:57:49.423842 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.423704    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2" event={"ID":"0d0015e3-f2b6-4d37-9f3c-b183498228fd","Type":"ContainerDied","Data":"6d656564b9cc71a9328690cda28441ef0057686c78906810677e85ce6b4a568c"}
Oct 30 21:57:49.433429 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.433407    4282 scope.go:122] "RemoveContainer" containerID="ee9f1bee0a58a4a254b67b12989e177c148ffa486bd068fce1ff38757ccc5517"
Oct 30 21:57:49.447913 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.447891    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"]
Oct 30 21:57:49.452397 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.452374    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-03f0a0-7cfdb549f-9v285"]
Oct 30 21:57:49.466998 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.466974    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a29d12-2b5f-43a4-a00b-07e87430976b" path="/var/lib/kubelet/pods/08a29d12-2b5f-43a4-a00b-07e87430976b/volumes"
Oct 30 21:57:49.467370 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.467355    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"]
Oct 30 21:57:49.467413 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:49.467390    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-03f0a0-6776bc7f-vqzg2"]
Oct 30 21:57:51.464983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:57:51.464954    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0015e3-f2b6-4d37-9f3c-b183498228fd" path="/var/lib/kubelet/pods/0d0015e3-f2b6-4d37-9f3c-b183498228fd/volumes"
Oct 30 21:58:01.636350 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:01.636313    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"]
Oct 30 21:58:01.650713 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:01.650682    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"]
Oct 30 21:58:04.659789 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:04.659743    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:58:06.637033 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:06.636992    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq" podUID="9a7d7e86-f143-4876-a5a5-2175ec447a33" containerName="payload-processing" containerID="cri-o://dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f" gracePeriod=25
Oct 30 21:58:06.651614 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:06.651568    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277" podUID="12c229a2-f5f0-49d7-8812-6165d0efb27b" containerName="payload-pre-processing" containerID="cri-o://1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7" gracePeriod=25
Oct 30 21:58:06.938253 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:06.938228    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:58:06.960712 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:06.960686    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:58:07.044335 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.044300    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c229a2-f5f0-49d7-8812-6165d0efb27b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c229a2-f5f0-49d7-8812-6165d0efb27b-plugins-config-volume\") pod \"12c229a2-f5f0-49d7-8812-6165d0efb27b\" (UID: \"12c229a2-f5f0-49d7-8812-6165d0efb27b\") "
Oct 30 21:58:07.044624 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.044604    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7d7e86-f143-4876-a5a5-2175ec447a33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d7e86-f143-4876-a5a5-2175ec447a33-plugins-config-volume\") pod \"9a7d7e86-f143-4876-a5a5-2175ec447a33\" (UID: \"9a7d7e86-f143-4876-a5a5-2175ec447a33\") "
Oct 30 21:58:07.044699 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.044679    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c229a2-f5f0-49d7-8812-6165d0efb27b-plugins-config-volume" pod "12c229a2-f5f0-49d7-8812-6165d0efb27b" (UID: "12c229a2-f5f0-49d7-8812-6165d0efb27b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:07.044699 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.044689    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c229a2-f5f0-49d7-8812-6165d0efb27b-kube-api-access-xwskq\" (UniqueName: \"kubernetes.io/projected/12c229a2-f5f0-49d7-8812-6165d0efb27b-kube-api-access-xwskq\") pod \"12c229a2-f5f0-49d7-8812-6165d0efb27b\" (UID: \"12c229a2-f5f0-49d7-8812-6165d0efb27b\") "
Oct 30 21:58:07.044823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.044762    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7d7e86-f143-4876-a5a5-2175ec447a33-kube-api-access-bv9qx\" (UniqueName: \"kubernetes.io/projected/9a7d7e86-f143-4876-a5a5-2175ec447a33-kube-api-access-bv9qx\") pod \"9a7d7e86-f143-4876-a5a5-2175ec447a33\" (UID: \"9a7d7e86-f143-4876-a5a5-2175ec447a33\") "
Oct 30 21:58:07.044998 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.044976    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7d7e86-f143-4876-a5a5-2175ec447a33-plugins-config-volume" pod "9a7d7e86-f143-4876-a5a5-2175ec447a33" (UID: "9a7d7e86-f143-4876-a5a5-2175ec447a33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:07.045423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.045403    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12c229a2-f5f0-49d7-8812-6165d0efb27b-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:07.045423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.045424    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7d7e86-f143-4876-a5a5-2175ec447a33-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:07.046948 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.046924    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c229a2-f5f0-49d7-8812-6165d0efb27b-kube-api-access-xwskq" pod "12c229a2-f5f0-49d7-8812-6165d0efb27b" (UID: "12c229a2-f5f0-49d7-8812-6165d0efb27b"). InnerVolumeSpecName "kube-api-access-xwskq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:07.047032 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.046976    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7d7e86-f143-4876-a5a5-2175ec447a33-kube-api-access-bv9qx" pod "9a7d7e86-f143-4876-a5a5-2175ec447a33" (UID: "9a7d7e86-f143-4876-a5a5-2175ec447a33"). InnerVolumeSpecName "kube-api-access-bv9qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:07.167551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.163742    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9qx\" (UniqueName: \"kubernetes.io/projected/9a7d7e86-f143-4876-a5a5-2175ec447a33-kube-api-access-bv9qx\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:07.167551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.163779    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwskq\" (UniqueName: \"kubernetes.io/projected/12c229a2-f5f0-49d7-8812-6165d0efb27b-kube-api-access-xwskq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:07.606540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.606508    4282 generic.go:356] "Generic (PLEG): container finished" podID="12c229a2-f5f0-49d7-8812-6165d0efb27b" containerID="1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7" exitCode=0
Oct 30 21:58:07.606689 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.606579    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277" event={"ID":"12c229a2-f5f0-49d7-8812-6165d0efb27b","Type":"ContainerDied","Data":"1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7"}
Oct 30 21:58:07.606689 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.606591    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"
Oct 30 21:58:07.606689 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.606636    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277" event={"ID":"12c229a2-f5f0-49d7-8812-6165d0efb27b","Type":"ContainerDied","Data":"d24b7543a1d5b344019030b0d2e68bfcd6b95e6003679c4419bc9deaf615efec"}
Oct 30 21:58:07.606689 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.606659    4282 scope.go:122] "RemoveContainer" containerID="1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7"
Oct 30 21:58:07.608105 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.608084    4282 generic.go:356] "Generic (PLEG): container finished" podID="9a7d7e86-f143-4876-a5a5-2175ec447a33" containerID="dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f" exitCode=0
Oct 30 21:58:07.608228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.608127    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"
Oct 30 21:58:07.608228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.608175    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq" event={"ID":"9a7d7e86-f143-4876-a5a5-2175ec447a33","Type":"ContainerDied","Data":"dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f"}
Oct 30 21:58:07.608228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.608217    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq" event={"ID":"9a7d7e86-f143-4876-a5a5-2175ec447a33","Type":"ContainerDied","Data":"d6d16d94da6c0afdcbcdf6e527e059b6599d173cb939e7570809483b64fc5c1d"}
Oct 30 21:58:07.616673 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.616656    4282 scope.go:122] "RemoveContainer" containerID="1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7"
Oct 30 21:58:07.616943 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:07.616925    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7\": container with ID starting with 1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7 not found: ID does not exist" containerID="1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7"
Oct 30 21:58:07.616992 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.616951    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7"} err="failed to get container status \"1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7\": rpc error: code = NotFound desc = could not find container \"1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7\": container with ID starting with 1e347f059923b4451d34c7d670e5695f8285599d4ad921b749a3fbfe94137ae7 not found: ID does not exist"
Oct 30 21:58:07.616992 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.616971    4282 scope.go:122] "RemoveContainer" containerID="dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f"
Oct 30 21:58:07.626548 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.626532    4282 scope.go:122] "RemoveContainer" containerID="dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f"
Oct 30 21:58:07.626829 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:07.626812    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f\": container with ID starting with dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f not found: ID does not exist" containerID="dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f"
Oct 30 21:58:07.626874 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.626837    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f"} err="failed to get container status \"dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f\": rpc error: code = NotFound desc = could not find container \"dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f\": container with ID starting with dd5346384e3d6f945e445603d3604aaa50684a0c69d6361ff837e84bfd4e816f not found: ID does not exist"
Oct 30 21:58:07.631857 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.631837    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"]
Oct 30 21:58:07.641344 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.641323    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ee0c56-5f8c877f79-7zvdq"]
Oct 30 21:58:07.651650 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.651629    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"]
Oct 30 21:58:07.669870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.669851    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ee0c56-5c548fccff-8r277"]
Oct 30 21:58:07.834656 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:07.834583    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:58:09.470556 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:09.470513    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c229a2-f5f0-49d7-8812-6165d0efb27b" path="/var/lib/kubelet/pods/12c229a2-f5f0-49d7-8812-6165d0efb27b/volumes"
Oct 30 21:58:09.471225 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:09.471197    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7d7e86-f143-4876-a5a5-2175ec447a33" path="/var/lib/kubelet/pods/9a7d7e86-f143-4876-a5a5-2175ec447a33/volumes"
Oct 30 21:58:12.361748 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:12.358982    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:12.509762 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:12.485802    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:12.509988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:12.508920    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 21:58:12.703455 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:12.703391    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:15.205022 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.204989    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"]
Oct 30 21:58:15.205853 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205830    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c42d6a09-5bf9-4e10-8576-44ad7442e7f0" containerName="payload-pre-processing"
Oct 30 21:58:15.205953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205856    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42d6a09-5bf9-4e10-8576-44ad7442e7f0" containerName="payload-pre-processing"
Oct 30 21:58:15.205953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205894    4282 cpu_manager.go:414] "containerMap: removing container" podUID="08a29d12-2b5f-43a4-a00b-07e87430976b" containerName="payload-pre-processing"
Oct 30 21:58:15.205953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205903    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a29d12-2b5f-43a4-a00b-07e87430976b" containerName="payload-pre-processing"
Oct 30 21:58:15.205953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205930    4282 cpu_manager.go:414] "containerMap: removing container" podUID="12c229a2-f5f0-49d7-8812-6165d0efb27b" containerName="payload-pre-processing"
Oct 30 21:58:15.205953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205938    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c229a2-f5f0-49d7-8812-6165d0efb27b" containerName="payload-pre-processing"
Oct 30 21:58:15.205953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205950    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0d0015e3-f2b6-4d37-9f3c-b183498228fd" containerName="payload-processing"
Oct 30 21:58:15.206219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205956    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0015e3-f2b6-4d37-9f3c-b183498228fd" containerName="payload-processing"
Oct 30 21:58:15.206219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205984    4282 cpu_manager.go:414] "containerMap: removing container" podUID="9a7d7e86-f143-4876-a5a5-2175ec447a33" containerName="payload-processing"
Oct 30 21:58:15.206219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.205992    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7d7e86-f143-4876-a5a5-2175ec447a33" containerName="payload-processing"
Oct 30 21:58:15.206219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.206124    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42d6a09-5bf9-4e10-8576-44ad7442e7f0" containerName="payload-pre-processing"
Oct 30 21:58:15.206219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.206141    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a29d12-2b5f-43a4-a00b-07e87430976b" containerName="payload-pre-processing"
Oct 30 21:58:15.206219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.206150    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c229a2-f5f0-49d7-8812-6165d0efb27b" containerName="payload-pre-processing"
Oct 30 21:58:15.206219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.206157    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0015e3-f2b6-4d37-9f3c-b183498228fd" containerName="payload-processing"
Oct 30 21:58:15.206219 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.206166    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7d7e86-f143-4876-a5a5-2175ec447a33" containerName="payload-processing"
Oct 30 21:58:15.211154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.211131    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.215069 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.215049    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3654d5-openshift-default-dockercfg-h757h\""
Oct 30 21:58:15.227936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.227907    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"]
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.348673    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.348753    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.348825    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnv9x\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-kube-api-access-qnv9x\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.348863    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fb080c-08ed-4eee-bcd1-893e4c192f69-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.349009    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.349033    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.349067    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.349192    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.350497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.349337    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.473604 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.473513    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.473870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.473851    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnv9x\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-kube-api-access-qnv9x\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.474002 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.473988    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fb080c-08ed-4eee-bcd1-893e4c192f69-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.474230 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.474215    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.474336 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.474323    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.474446 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.474433    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.474663 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.474648    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.474869 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.474855    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.487494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.475009    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.487865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.480889    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.488010 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.481431    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.488122 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.481684    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.488238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.481907    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.488333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.482500    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fb080c-08ed-4eee-bcd1-893e4c192f69-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.549491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.548356    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnv9x\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-kube-api-access-qnv9x\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.549491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.549078    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.549491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.549374    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.557029 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.557001    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:15.858551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:15.856787    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:16.278953 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:16.278927    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"]
Oct 30 21:58:16.281202 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:58:16.281174    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fb080c_08ed_4eee_bcd1_893e4c192f69.slice/crio-28fac0ed86de509c27306b4f9bf1bcfe7c7240bc940502b5ea095137f3e7d257 WatchSource:0}: Error finding container 28fac0ed86de509c27306b4f9bf1bcfe7c7240bc940502b5ea095137f3e7d257: Status 404 returned error can't find the container with id 28fac0ed86de509c27306b4f9bf1bcfe7c7240bc940502b5ea095137f3e7d257
Oct 30 21:58:16.283696 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:16.283655    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:16.283794 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:16.283736    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:16.283794 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:16.283770    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:16.972493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:16.971617    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" event={"ID":"81fb080c-08ed-4eee-bcd1-893e4c192f69","Type":"ContainerStarted","Data":"17c232b350d54e703c535f68a4f0c21f828efc255047d1fd6bb4a9d6db547ee1"}
Oct 30 21:58:16.972493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:16.971687    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" event={"ID":"81fb080c-08ed-4eee-bcd1-893e4c192f69","Type":"ContainerStarted","Data":"28fac0ed86de509c27306b4f9bf1bcfe7c7240bc940502b5ea095137f3e7d257"}
Oct 30 21:58:17.006862 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:17.006814    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podStartSLOduration=2.006798935 podStartE2EDuration="2.006798935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:17.00514703 +0000 UTC m=+1115.663243267" watchObservedRunningTime="2026-10-30 21:58:17.006798935 +0000 UTC m=+1115.664895225"
Oct 30 21:58:17.778230 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:17.778191    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"]
Oct 30 21:58:17.779118 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:17.779085    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx" podUID="b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" containerName="maas-api" containerID="cri-o://60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522" gracePeriod=30
Oct 30 21:58:17.884083 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:17.861715    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:18.102005 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.101923    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:18.547983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.547956    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:58:18.606185 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.606148    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-maas-api-tls\") pod \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") "
Oct 30 21:58:18.625430 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.625398    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-kube-api-access-4v2dh\" (UniqueName: \"kubernetes.io/projected/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-kube-api-access-4v2dh\") pod \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") "
Oct 30 21:58:18.625749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.625729    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-openshift-service-ca\") pod \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") "
Oct 30 21:58:18.625908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.625891    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-metrics-certs\") pod \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\" (UID: \"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa\") "
Oct 30 21:58:18.626166 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.626116    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-openshift-service-ca" pod "b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" (UID: "b0df5f56-45ee-42c3-9590-c0fdd7d56cfa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:18.631984 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.631946    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:18.642123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.642053    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-maas-api-tls" pod "b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" (UID: "b0df5f56-45ee-42c3-9590-c0fdd7d56cfa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:18.646021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.645859    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-metrics-certs" pod "b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" (UID: "b0df5f56-45ee-42c3-9590-c0fdd7d56cfa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:18.674636 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.674536    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-kube-api-access-4v2dh" pod "b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" (UID: "b0df5f56-45ee-42c3-9590-c0fdd7d56cfa"). InnerVolumeSpecName "kube-api-access-4v2dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:18.788589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.779096    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v2dh\" (UniqueName: \"kubernetes.io/projected/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-kube-api-access-4v2dh\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:18.788589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.779133    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:18.788589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:18.779149    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:19.074135 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.074052    4282 generic.go:356] "Generic (PLEG): container finished" podID="b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" containerID="60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522" exitCode=0
Oct 30 21:58:19.074135 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.074100    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"
Oct 30 21:58:19.074333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.074132    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx" event={"ID":"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa","Type":"ContainerDied","Data":"60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522"}
Oct 30 21:58:19.074333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.074184    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx" event={"ID":"b0df5f56-45ee-42c3-9590-c0fdd7d56cfa","Type":"ContainerDied","Data":"a6805c58ed5e7916dd40bbe1ba3cb54955f9c19d3510af5665af8207a8274510"}
Oct 30 21:58:19.074333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.074214    4282 scope.go:122] "RemoveContainer" containerID="60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522"
Oct 30 21:58:19.074704 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.074683    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:19.075871 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.075851    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:58:19.085682 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.085659    4282 scope.go:122] "RemoveContainer" containerID="60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522"
Oct 30 21:58:19.085969 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:19.085943    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522\": container with ID starting with 60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522 not found: ID does not exist" containerID="60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522"
Oct 30 21:58:19.086026 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.085980    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522"} err="failed to get container status \"60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522\": rpc error: code = NotFound desc = could not find container \"60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522\": container with ID starting with 60f719093429c7a7489501eced17654a296b0c48a6666d9c2c383dfa5e64d522 not found: ID does not exist"
Oct 30 21:58:19.224117 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.219769    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"]
Oct 30 21:58:19.266258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.247374    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ee0c56-75776447b8-zqswx"]
Oct 30 21:58:19.465098 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:19.465064    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" path="/var/lib/kubelet/pods/b0df5f56-45ee-42c3-9590-c0fdd7d56cfa/volumes"
Oct 30 21:58:22.029389 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.029355    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"]
Oct 30 21:58:22.037527 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.037436    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd" podUID="1c6cc795-b6a1-4a4a-80d4-02645ee38f27" containerName="maas-api" containerID="cri-o://9365779ffdc186a1fb3067e7f16af7e4d377ea836d63b51a37dce176004f64ef" gracePeriod=30
Oct 30 21:58:22.193666 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.193621    4282 generic.go:356] "Generic (PLEG): container finished" podID="1c6cc795-b6a1-4a4a-80d4-02645ee38f27" containerID="9365779ffdc186a1fb3067e7f16af7e4d377ea836d63b51a37dce176004f64ef" exitCode=0
Oct 30 21:58:22.193833 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.193666    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd" event={"ID":"1c6cc795-b6a1-4a4a-80d4-02645ee38f27","Type":"ContainerDied","Data":"9365779ffdc186a1fb3067e7f16af7e4d377ea836d63b51a37dce176004f64ef"}
Oct 30 21:58:22.815813 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.815779    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:58:22.871967 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.871931    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-openshift-service-ca\") pod \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") "
Oct 30 21:58:22.872149 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.872032    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-kube-api-access-8hz9c\" (UniqueName: \"kubernetes.io/projected/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-kube-api-access-8hz9c\") pod \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") "
Oct 30 21:58:22.872238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.872215    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-maas-api-tls\") pod \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") "
Oct 30 21:58:22.872352 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.872330    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-openshift-service-ca" pod "1c6cc795-b6a1-4a4a-80d4-02645ee38f27" (UID: "1c6cc795-b6a1-4a4a-80d4-02645ee38f27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:22.872410 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.872380    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-metrics-certs\") pod \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\" (UID: \"1c6cc795-b6a1-4a4a-80d4-02645ee38f27\") "
Oct 30 21:58:22.873082 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.873055    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:22.875602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.875572    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-maas-api-tls" pod "1c6cc795-b6a1-4a4a-80d4-02645ee38f27" (UID: "1c6cc795-b6a1-4a4a-80d4-02645ee38f27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:22.876118 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.876093    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-kube-api-access-8hz9c" pod "1c6cc795-b6a1-4a4a-80d4-02645ee38f27" (UID: "1c6cc795-b6a1-4a4a-80d4-02645ee38f27"). InnerVolumeSpecName "kube-api-access-8hz9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:22.876328 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.876302    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-metrics-certs" pod "1c6cc795-b6a1-4a4a-80d4-02645ee38f27" (UID: "1c6cc795-b6a1-4a4a-80d4-02645ee38f27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:22.974513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.974481    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hz9c\" (UniqueName: \"kubernetes.io/projected/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-kube-api-access-8hz9c\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:22.974513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.974507    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:22.974513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:22.974516    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6cc795-b6a1-4a4a-80d4-02645ee38f27-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:23.205274 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:23.205238    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd" event={"ID":"1c6cc795-b6a1-4a4a-80d4-02645ee38f27","Type":"ContainerDied","Data":"4a27ca9e929c76e4abbe6b3e9638927a927ea3e1c24731a01d92f1da00529403"}
Oct 30 21:58:23.205821 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:23.205278    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"
Oct 30 21:58:23.205821 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:23.205310    4282 scope.go:122] "RemoveContainer" containerID="9365779ffdc186a1fb3067e7f16af7e4d377ea836d63b51a37dce176004f64ef"
Oct 30 21:58:23.236549 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:23.236518    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"]
Oct 30 21:58:23.244081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:23.244055    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd"]
Oct 30 21:58:23.465337 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:23.465238    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6cc795-b6a1-4a4a-80d4-02645ee38f27" path="/var/lib/kubelet/pods/1c6cc795-b6a1-4a4a-80d4-02645ee38f27/volumes"
Oct 30 21:58:24.213140 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:24.213065    4282 generic.go:356] "Generic (PLEG): container finished" podID="8f390701-2496-414e-ada8-90201df827ae" containerID="eb407652e7ef23c3b7f08322a78a703d20c3f8f234ac63ab7a16b750d7c8a295" exitCode=0
Oct 30 21:58:24.213657 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:24.213136    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" event={"ID":"8f390701-2496-414e-ada8-90201df827ae","Type":"ContainerDied","Data":"eb407652e7ef23c3b7f08322a78a703d20c3f8f234ac63ab7a16b750d7c8a295"}
Oct 30 21:58:24.213657 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:24.213606    4282 scope.go:122] "RemoveContainer" containerID="eb407652e7ef23c3b7f08322a78a703d20c3f8f234ac63ab7a16b750d7c8a295"
Oct 30 21:58:24.215070 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:24.215043    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:24.215143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:24.215109    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:24.215179 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:24.215142    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:25.225610 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:25.225578    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" event={"ID":"8f390701-2496-414e-ada8-90201df827ae","Type":"ContainerStarted","Data":"30bf5eb59f978473e87340cd40f908b666f25a0342a3d18e86536b0765d86d3e"}
Oct 30 21:58:25.372494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:25.371596    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:58:25.668143 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:25.668118    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:58:25.686547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:25.668540    4282 patch_prober.go:29] interesting pod/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.686912 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:25.686876    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:58:25.801758 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:25.801717    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-56c0af-7d4bc76d96-kglzd" podUID="1c6cc795-b6a1-4a4a-80d4-02645ee38f27" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.91:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:26.317495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.314525    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"]
Oct 30 21:58:26.317495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.315521    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" containerName="maas-api"
Oct 30 21:58:26.317495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.315544    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" containerName="maas-api"
Oct 30 21:58:26.317495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.315569    4282 cpu_manager.go:414] "containerMap: removing container" podUID="1c6cc795-b6a1-4a4a-80d4-02645ee38f27" containerName="maas-api"
Oct 30 21:58:26.317495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.315579    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6cc795-b6a1-4a4a-80d4-02645ee38f27" containerName="maas-api"
Oct 30 21:58:26.317495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.315796    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0df5f56-45ee-42c3-9590-c0fdd7d56cfa" containerName="maas-api"
Oct 30 21:58:26.317495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.315814    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6cc795-b6a1-4a4a-80d4-02645ee38f27" containerName="maas-api"
Oct 30 21:58:26.333493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.325098    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.338491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.334611    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"]
Oct 30 21:58:26.349891 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.345749    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3654d5-serving-cert\""
Oct 30 21:58:26.531499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.529492    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8szrz\" (UniqueName: \"kubernetes.io/projected/2d0b762c-db09-4e87-9021-5e34a42eb86e-kube-api-access-8szrz\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.531499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.529702    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0b762c-db09-4e87-9021-5e34a42eb86e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.531499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.529769    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.531499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.529890    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-metrics-certs\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.670280 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.670240    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0b762c-db09-4e87-9021-5e34a42eb86e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.670639 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.670603    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.670931 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.670914    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-metrics-certs\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.671270 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.671250    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8szrz\" (UniqueName: \"kubernetes.io/projected/2d0b762c-db09-4e87-9021-5e34a42eb86e-kube-api-access-8szrz\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.694769 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:26.672777    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-serving-cert: secret "maas-api-e2e-ipp-a-3654d5-serving-cert" not found
Oct 30 21:58:26.695483 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:26.695442    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls podName:2d0b762c-db09-4e87-9021-5e34a42eb86e nodeName:}" failed. No retries permitted until 2026-10-30 21:58:27.195404631 +0000 UTC m=+1125.853500846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls") pod "maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh" (UID: "2d0b762c-db09-4e87-9021-5e34a42eb86e") : secret "maas-api-e2e-ipp-a-3654d5-serving-cert" not found
Oct 30 21:58:26.723618 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.723404    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0b762c-db09-4e87-9021-5e34a42eb86e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.726946 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.726922    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-metrics-certs\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:26.733324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.729135    4282 patch_prober.go:29] interesting pod/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:58:26.733324 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.731194    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:58:26.746555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:26.746487    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8szrz\" (UniqueName: \"kubernetes.io/projected/2d0b762c-db09-4e87-9021-5e34a42eb86e-kube-api-access-8szrz\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:27.286620 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:27.286586    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:27.289300 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:27.289275    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:27.526168 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:27.525906    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:27.684177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:27.683706    4282 patch_prober.go:29] interesting pod/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:58:27.684177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:27.683772    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:58:27.933931 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:27.933901    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"]
Oct 30 21:58:27.934954 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:58:27.934926    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0b762c_db09_4e87_9021_5e34a42eb86e.slice/crio-2d0298c97d8e12a3499a992a5136e8ae36403009ab42d5ba79e2e256cdf00d81 WatchSource:0}: Error finding container 2d0298c97d8e12a3499a992a5136e8ae36403009ab42d5ba79e2e256cdf00d81: Status 404 returned error can't find the container with id 2d0298c97d8e12a3499a992a5136e8ae36403009ab42d5ba79e2e256cdf00d81
Oct 30 21:58:28.291003 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:28.290914    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh" event={"ID":"2d0b762c-db09-4e87-9021-5e34a42eb86e","Type":"ContainerStarted","Data":"2d0298c97d8e12a3499a992a5136e8ae36403009ab42d5ba79e2e256cdf00d81"}
Oct 30 21:58:28.718702 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:28.718566    4282 patch_prober.go:29] interesting pod/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:58:28.719788 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:28.719755    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:58:29.300172 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:29.300133    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh" event={"ID":"2d0b762c-db09-4e87-9021-5e34a42eb86e","Type":"ContainerStarted","Data":"e2296cd644fd51e18bf62b3233ca586139b0413257bb5886d49b20ce3109dc02"}
Oct 30 21:58:29.300363 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:29.300252    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:29.333551 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:29.333505    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh" podStartSLOduration=2.590126782 podStartE2EDuration="3.333492393s" totalImagesPullingTime="743.365611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:29.329844189 +0000 UTC m=+1127.987940412" watchObservedRunningTime="2026-10-30 21:58:29.333492393 +0000 UTC m=+1127.991588613"
Oct 30 21:58:29.678864 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:29.678833    4282 patch_prober.go:29] interesting pod/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:58:29.679142 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:29.679113    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:58:30.670884 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:30.670851    4282 patch_prober.go:29] interesting pod/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 21:58:30.671501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:30.671449    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 21:58:31.265627 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.265587    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"]
Oct 30 21:58:31.271377 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.271356    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.288079 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.288050    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"]
Oct 30 21:58:31.292827 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.292798    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f47f79-openshift-default-dockercfg-vldts\""
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.426856    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-envoy\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.426947    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-socket\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.426973    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-data\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.427076    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78xn\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-kube-api-access-m78xn\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.427124    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68171810-3eb8-4c3a-a758-f11eaafe5082-istiod-ca-cert\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.427157    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-certs\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.427261    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-token\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.427312    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-credential-socket\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.427369    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-podinfo\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.536547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.534122    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68171810-3eb8-4c3a-a758-f11eaafe5082-istiod-ca-cert\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.536547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.534175    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-certs\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.536547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.534268    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-token\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.536547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.534311    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-credential-socket\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.536547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.534360    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-podinfo\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.548497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.542721    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-certs\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.548497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.543373    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68171810-3eb8-4c3a-a758-f11eaafe5082-istiod-ca-cert\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.548497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.543975    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-credential-socket\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.548497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.544078    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-envoy\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.548497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.544169    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-socket\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.548497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.544194    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-data\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.548497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.544317    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78xn\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-kube-api-access-m78xn\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.550893 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.550861    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-socket\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.561409 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.561382    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-data\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.574494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.570149    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78xn\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-kube-api-access-m78xn\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.574494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.570804    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-envoy\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.574494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.571345    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-podinfo\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.574494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.572776    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-token\") pod \"e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") " pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.607504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.603841    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:31.864665 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.863110    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:58:31.869201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:31.868784    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:58:32.143548 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:32.143452    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"]
Oct 30 21:58:32.145315 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:32.145272    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:32.145421 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:32.145367    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:32.145421 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:32.145414    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:32.342003 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:32.341971    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" event={"ID":"68171810-3eb8-4c3a-a758-f11eaafe5082","Type":"ContainerStarted","Data":"0cb9ee38f08ed3ba3961543353b0e79586b80a61619fd12b472b1e0b86262916"}
Oct 30 21:58:32.609989 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:32.609958    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"]
Oct 30 21:58:33.350013 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:33.349980    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" event={"ID":"68171810-3eb8-4c3a-a758-f11eaafe5082","Type":"ContainerStarted","Data":"b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18"}
Oct 30 21:58:33.350457 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:33.350076    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy" containerID="cri-o://30bf5eb59f978473e87340cd40f908b666f25a0342a3d18e86536b0765d86d3e" gracePeriod=30
Oct 30 21:58:33.413310 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:33.413259    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" podStartSLOduration=2.413244306 podStartE2EDuration="2.413244306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:33.410576789 +0000 UTC m=+1132.068673012" watchObservedRunningTime="2026-10-30 21:58:33.413244306 +0000 UTC m=+1132.071340529"
Oct 30 21:58:33.611617 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:33.606874    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:33.628070 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:33.627325    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:34.387494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:34.380542    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:34.387494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:34.381789    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:58:34.683161 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:34.683123    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:58:35.313665 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:35.313629    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 21:58:38.481136 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.481109    4282 generic.go:356] "Generic (PLEG): container finished" podID="8f390701-2496-414e-ada8-90201df827ae" containerID="30bf5eb59f978473e87340cd40f908b666f25a0342a3d18e86536b0765d86d3e" exitCode=0
Oct 30 21:58:38.481609 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.481182    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" event={"ID":"8f390701-2496-414e-ada8-90201df827ae","Type":"ContainerDied","Data":"30bf5eb59f978473e87340cd40f908b666f25a0342a3d18e86536b0765d86d3e"}
Oct 30 21:58:38.481609 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.481290    4282 scope.go:122] "RemoveContainer" containerID="eb407652e7ef23c3b7f08322a78a703d20c3f8f234ac63ab7a16b750d7c8a295"
Oct 30 21:58:38.627025 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.626971    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:58:38.684893 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.684855    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-certs\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685153    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-certs" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:38.685385 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685298    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f390701-2496-414e-ada8-90201df827ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f390701-2496-414e-ada8-90201df827ae-istio-podinfo\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685385 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685338    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-data\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685388    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-envoy\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685445    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f390701-2496-414e-ada8-90201df827ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f390701-2496-414e-ada8-90201df827ae-istiod-ca-cert\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685520    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-credential-socket\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685722 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685626    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-socket\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685722 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685640    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-data" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:38.685722 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685696    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-istio-token\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-istio-token\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685815    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-kube-api-access-gh82j\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-kube-api-access-gh82j\") pod \"8f390701-2496-414e-ada8-90201df827ae\" (UID: \"8f390701-2496-414e-ada8-90201df827ae\") "
Oct 30 21:58:38.685947 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.685929    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f390701-2496-414e-ada8-90201df827ae-istiod-ca-cert" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:38.686851 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.686217    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-socket" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:38.686851 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.686566    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f390701-2496-414e-ada8-90201df827ae-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.686851 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.686589    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.686851 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.686602    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.686851 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.686616    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.687316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.687280    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-credential-socket" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:38.690154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.690127    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-envoy" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:38.690154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.690135    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-kube-api-access-gh82j" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "kube-api-access-gh82j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:38.690325 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.690200    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f390701-2496-414e-ada8-90201df827ae-istio-podinfo" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:58:38.690536 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.690458    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-istio-token" pod "8f390701-2496-414e-ada8-90201df827ae" (UID: "8f390701-2496-414e-ada8-90201df827ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:38.788264 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.788237    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.788264 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.788265    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.788452 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.788279    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh82j\" (UniqueName: \"kubernetes.io/projected/8f390701-2496-414e-ada8-90201df827ae-kube-api-access-gh82j\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.788452 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.788293    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f390701-2496-414e-ada8-90201df827ae-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.788452 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.788304    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f390701-2496-414e-ada8-90201df827ae-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:38.871951 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.871909    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"]
Oct 30 21:58:38.901566 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:38.901438    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"]
Oct 30 21:58:39.547194 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.547145    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5" event={"ID":"8f390701-2496-414e-ada8-90201df827ae","Type":"ContainerDied","Data":"d497f860c5f474d834bf6002411fb2ea37aba809ae21844375a66d848f82ef49"}
Oct 30 21:58:39.547775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.547228    4282 scope.go:122] "RemoveContainer" containerID="30bf5eb59f978473e87340cd40f908b666f25a0342a3d18e86536b0765d86d3e"
Oct 30 21:58:39.547775 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.547353    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"
Oct 30 21:58:39.592440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.592407    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"]
Oct 30 21:58:39.654490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.653827    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56c0af-openshift-default-855b44465c-k9bx5"]
Oct 30 21:58:39.924260 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.924230    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"]
Oct 30 21:58:39.924874 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.924859    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy"
Oct 30 21:58:39.924936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.924876    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy"
Oct 30 21:58:39.924936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.924905    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy"
Oct 30 21:58:39.924936 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.924910    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy"
Oct 30 21:58:39.925028 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.925017    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy"
Oct 30 21:58:39.929797 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.929778    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:39.939756 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.939733    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0b3fe4-dockercfg-l6x8f\""
Oct 30 21:58:39.941967 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.941943    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"]
Oct 30 21:58:39.957608 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.957582    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zp5p\" (UniqueName: \"kubernetes.io/projected/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-kube-api-access-6zp5p\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5\" (UID: \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:39.957740 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.957671    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5\" (UID: \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:39.999631 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:39.999601    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"]
Oct 30 21:58:40.000298 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.000284    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f390701-2496-414e-ada8-90201df827ae" containerName="istio-proxy"
Oct 30 21:58:40.003838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.003819    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:40.023059 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.023038    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0b3fe4\""
Oct 30 21:58:40.023888 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.023869    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"]
Oct 30 21:58:40.113495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.100210    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnvw\" (UniqueName: \"kubernetes.io/projected/5bbbb8f8-acdb-4fe8-9fac-88839b560961-kube-api-access-bbnvw\") pod \"payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk\" (UID: \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:40.113495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.100361    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zp5p\" (UniqueName: \"kubernetes.io/projected/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-kube-api-access-6zp5p\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5\" (UID: \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:40.113495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.100515    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5\" (UID: \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:40.113495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.100570    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbbb8f8-acdb-4fe8-9fac-88839b560961-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk\" (UID: \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:40.136517 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.132025    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5\" (UID: \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:40.172516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.159029    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zp5p\" (UniqueName: \"kubernetes.io/projected/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-kube-api-access-6zp5p\") pod \"payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5\" (UID: \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:40.205544 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.205139    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnvw\" (UniqueName: \"kubernetes.io/projected/5bbbb8f8-acdb-4fe8-9fac-88839b560961-kube-api-access-bbnvw\") pod \"payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk\" (UID: \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:40.205544 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.205355    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbbb8f8-acdb-4fe8-9fac-88839b560961-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk\" (UID: \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:40.214003 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.206294    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbbb8f8-acdb-4fe8-9fac-88839b560961-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk\" (UID: \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:40.283493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.271906    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:40.349494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.341130    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnvw\" (UniqueName: \"kubernetes.io/projected/5bbbb8f8-acdb-4fe8-9fac-88839b560961-kube-api-access-bbnvw\") pod \"payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk\" (UID: \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:40.617662 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.617587    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:40.679900 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:40.679877    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"]
Oct 30 21:58:40.680218 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:58:40.680175    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1fd603_5d60_4e66_bd3b_1d67f1b34fc3.slice/crio-7452742d4bf880c2ab2de8bc45953e233fadb1ff7e3b9288af009a1ad40e91b2 WatchSource:0}: Error finding container 7452742d4bf880c2ab2de8bc45953e233fadb1ff7e3b9288af009a1ad40e91b2: Status 404 returned error can't find the container with id 7452742d4bf880c2ab2de8bc45953e233fadb1ff7e3b9288af009a1ad40e91b2
Oct 30 21:58:41.106192 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.100357    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"]
Oct 30 21:58:41.562692 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.562596    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f390701-2496-414e-ada8-90201df827ae" path="/var/lib/kubelet/pods/8f390701-2496-414e-ada8-90201df827ae/volumes"
Oct 30 21:58:41.563378 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.563358    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"]
Oct 30 21:58:41.570556 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.570531    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.571810 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.571792    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"]
Oct 30 21:58:41.578508 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.577216    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"]
Oct 30 21:58:41.578508 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.577430    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy" containerID="cri-o://f31f67b42b5e2659f32439918c3b0372d6559ae3751f6c992396fa1984774b0c" gracePeriod=30
Oct 30 21:58:41.579527 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.579312    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3e3670-openshift-default-dockercfg-vgt6t\""
Oct 30 21:58:41.590760 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.590619    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk" event={"ID":"5bbbb8f8-acdb-4fe8-9fac-88839b560961","Type":"ContainerStarted","Data":"caab1adff54a9fa6868a10883eeffe1ebca68385e3fcd0b413fda2b46efe2705"}
Oct 30 21:58:41.695368 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.694306    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5" event={"ID":"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3","Type":"ContainerStarted","Data":"a105b9679b8acc62934d2e016c088c82eea21b79700f1077a131c0516c3d97c9"}
Oct 30 21:58:41.695368 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.694376    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5" event={"ID":"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3","Type":"ContainerStarted","Data":"7452742d4bf880c2ab2de8bc45953e233fadb1ff7e3b9288af009a1ad40e91b2"}
Oct 30 21:58:41.695368 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.694447    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:41.805228 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.805191    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-credential-socket\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.805563 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.805539    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87lk9\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-kube-api-access-87lk9\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.805750 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.805732    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istiod-ca-cert\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.805873 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.805859    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-podinfo\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.805983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.805970    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-token\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.806148 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.806130    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-envoy\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.806251 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.806237    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-certs\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.806352 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.806340    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-data\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.809602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.809576    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-socket\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.834589 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.834447    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5" podStartSLOduration=2.834424588 podStartE2EDuration="2.834424588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:41.744565526 +0000 UTC m=+1140.402661752" watchObservedRunningTime="2026-10-30 21:58:41.834424588 +0000 UTC m=+1140.492520812"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.911418    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-socket\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.911743    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-credential-socket\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.911875    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87lk9\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-kube-api-access-87lk9\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.912086    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istiod-ca-cert\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.912125    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-podinfo\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.912159    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-token\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.912273    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-envoy\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.912304    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-certs\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.912348    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-data\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.912895    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-data\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.913153    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-socket\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.914097    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istiod-ca-cert\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.914495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.914349    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-credential-socket\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.918979 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.918780    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-certs\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.950848 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.934890    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-envoy\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.950848 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.940254    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-podinfo\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.983255 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.978707    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-token\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:41.985640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:41.985611    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87lk9\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-kube-api-access-87lk9\") pod \"e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") " pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:42.224883 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.224854    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3e3670-openshift-default-dockercfg-vgt6t\""
Oct 30 21:58:42.234018 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.233992    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:42.530554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.530530    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"]
Oct 30 21:58:42.530697 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:58:42.530679    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4a5ed4_af4c_4c26_8600_bae241eb1982.slice/crio-b0576ec7e956eecbd54d1c102e2a0cdbc86ffdf0ce579ba3e72ea58e6d925c65 WatchSource:0}: Error finding container b0576ec7e956eecbd54d1c102e2a0cdbc86ffdf0ce579ba3e72ea58e6d925c65: Status 404 returned error can't find the container with id b0576ec7e956eecbd54d1c102e2a0cdbc86ffdf0ce579ba3e72ea58e6d925c65
Oct 30 21:58:42.533249 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.533210    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:42.533316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.533287    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:42.533362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.533317    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:58:42.703058 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.703025    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk" event={"ID":"5bbbb8f8-acdb-4fe8-9fac-88839b560961","Type":"ContainerStarted","Data":"656809f6f3b84bca8bf2097ccf6b0337ff6dc199ed456c1c8335c194ebb4bc5c"}
Oct 30 21:58:42.703555 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.703119    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:42.706033 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.706003    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" event={"ID":"aa4a5ed4-af4c-4c26-8600-bae241eb1982","Type":"ContainerStarted","Data":"e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548"}
Oct 30 21:58:42.706177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.706057    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" event={"ID":"aa4a5ed4-af4c-4c26-8600-bae241eb1982","Type":"ContainerStarted","Data":"b0576ec7e956eecbd54d1c102e2a0cdbc86ffdf0ce579ba3e72ea58e6d925c65"}
Oct 30 21:58:42.801102 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.800743    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podStartSLOduration=1.8007231479999999 podStartE2EDuration="1.800723148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:42.799313444 +0000 UTC m=+1141.457409667" watchObservedRunningTime="2026-10-30 21:58:42.800723148 +0000 UTC m=+1141.458819406"
Oct 30 21:58:42.804354 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:42.804312    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk" podStartSLOduration=3.804298293 podStartE2EDuration="3.804298293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:42.742621466 +0000 UTC m=+1141.400717688" watchObservedRunningTime="2026-10-30 21:58:42.804298293 +0000 UTC m=+1141.462394519"
Oct 30 21:58:43.266492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:43.264584    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:43.277817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:43.275940    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:58:43.277817 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:43.275984    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 21:58:43.872502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:43.872444    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf" podUID="991c83df-52d4-4454-b757-27129f8f41c5" containerName="payload-processing" containerID="cri-o://2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc" gracePeriod=25
Oct 30 21:58:43.902422 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:43.902376    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p" podUID="7149abdb-cbb4-45d6-88d3-008adf585246" containerName="payload-pre-processing" containerID="cri-o://2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48" gracePeriod=25
Oct 30 21:58:44.188879 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.188849    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:58:44.229138 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.229113    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:58:44.240186 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.240162    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:44.301988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.301948    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991c83df-52d4-4454-b757-27129f8f41c5-kube-api-access-hr5gj\" (UniqueName: \"kubernetes.io/projected/991c83df-52d4-4454-b757-27129f8f41c5-kube-api-access-hr5gj\") pod \"991c83df-52d4-4454-b757-27129f8f41c5\" (UID: \"991c83df-52d4-4454-b757-27129f8f41c5\") "
Oct 30 21:58:44.302180 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.302092    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991c83df-52d4-4454-b757-27129f8f41c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/991c83df-52d4-4454-b757-27129f8f41c5-plugins-config-volume\") pod \"991c83df-52d4-4454-b757-27129f8f41c5\" (UID: \"991c83df-52d4-4454-b757-27129f8f41c5\") "
Oct 30 21:58:44.302744 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.302716    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991c83df-52d4-4454-b757-27129f8f41c5-plugins-config-volume" pod "991c83df-52d4-4454-b757-27129f8f41c5" (UID: "991c83df-52d4-4454-b757-27129f8f41c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:44.304787 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.304757    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991c83df-52d4-4454-b757-27129f8f41c5-kube-api-access-hr5gj" pod "991c83df-52d4-4454-b757-27129f8f41c5" (UID: "991c83df-52d4-4454-b757-27129f8f41c5"). InnerVolumeSpecName "kube-api-access-hr5gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:44.403262 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.403195    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7149abdb-cbb4-45d6-88d3-008adf585246-kube-api-access-zqvrk\" (UniqueName: \"kubernetes.io/projected/7149abdb-cbb4-45d6-88d3-008adf585246-kube-api-access-zqvrk\") pod \"7149abdb-cbb4-45d6-88d3-008adf585246\" (UID: \"7149abdb-cbb4-45d6-88d3-008adf585246\") "
Oct 30 21:58:44.403396 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.403348    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7149abdb-cbb4-45d6-88d3-008adf585246-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149abdb-cbb4-45d6-88d3-008adf585246-plugins-config-volume\") pod \"7149abdb-cbb4-45d6-88d3-008adf585246\" (UID: \"7149abdb-cbb4-45d6-88d3-008adf585246\") "
Oct 30 21:58:44.403735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.403711    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7149abdb-cbb4-45d6-88d3-008adf585246-plugins-config-volume" pod "7149abdb-cbb4-45d6-88d3-008adf585246" (UID: "7149abdb-cbb4-45d6-88d3-008adf585246"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:44.404069 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.404051    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5gj\" (UniqueName: \"kubernetes.io/projected/991c83df-52d4-4454-b757-27129f8f41c5-kube-api-access-hr5gj\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.404136 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.404073    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7149abdb-cbb4-45d6-88d3-008adf585246-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.404136 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.404087    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/991c83df-52d4-4454-b757-27129f8f41c5-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.405512 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.405487    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7149abdb-cbb4-45d6-88d3-008adf585246-kube-api-access-zqvrk" pod "7149abdb-cbb4-45d6-88d3-008adf585246" (UID: "7149abdb-cbb4-45d6-88d3-008adf585246"). InnerVolumeSpecName "kube-api-access-zqvrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:44.505797 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.505764    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqvrk\" (UniqueName: \"kubernetes.io/projected/7149abdb-cbb4-45d6-88d3-008adf585246-kube-api-access-zqvrk\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.727249 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.727210    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"]
Oct 30 21:58:44.742854 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.742711    4282 generic.go:356] "Generic (PLEG): container finished" podID="991c83df-52d4-4454-b757-27129f8f41c5" containerID="2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc" exitCode=0
Oct 30 21:58:44.743327 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.742994    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"
Oct 30 21:58:44.765499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.748350    4282 cpu_manager.go:414] "containerMap: removing container" podUID="991c83df-52d4-4454-b757-27129f8f41c5" containerName="payload-processing"
Oct 30 21:58:44.765749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.765722    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991c83df-52d4-4454-b757-27129f8f41c5" containerName="payload-processing"
Oct 30 21:58:44.765913 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.765899    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7149abdb-cbb4-45d6-88d3-008adf585246" containerName="payload-pre-processing"
Oct 30 21:58:44.766002 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.765990    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7149abdb-cbb4-45d6-88d3-008adf585246" containerName="payload-pre-processing"
Oct 30 21:58:44.789922 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.766330    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="991c83df-52d4-4454-b757-27129f8f41c5" containerName="payload-processing"
Oct 30 21:58:44.790297 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.790273    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7149abdb-cbb4-45d6-88d3-008adf585246" containerName="payload-pre-processing"
Oct 30 21:58:44.799533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.755388    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"
Oct 30 21:58:44.799533 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.755306    4282 generic.go:356] "Generic (PLEG): container finished" podID="7149abdb-cbb4-45d6-88d3-008adf585246" containerID="2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48" exitCode=0
Oct 30 21:58:44.825958 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.825923    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:44.826123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.826009    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:58:44.826123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.826033    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf" event={"ID":"991c83df-52d4-4454-b757-27129f8f41c5","Type":"ContainerDied","Data":"2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc"}
Oct 30 21:58:44.826123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.826076    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf" event={"ID":"991c83df-52d4-4454-b757-27129f8f41c5","Type":"ContainerDied","Data":"3551ffe9b617b95f99711ef146611332d36bbb1d32d65038ceb7193e0d22f480"}
Oct 30 21:58:44.826123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.826096    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p" event={"ID":"7149abdb-cbb4-45d6-88d3-008adf585246","Type":"ContainerDied","Data":"2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48"}
Oct 30 21:58:44.826123 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.826112    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p" event={"ID":"7149abdb-cbb4-45d6-88d3-008adf585246","Type":"ContainerDied","Data":"e269a8b504bc107f3010cd1274c2e90b5989ad4a556cca0dcd365bd29c4a6b4b"}
Oct 30 21:58:44.826382 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.826141    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"]
Oct 30 21:58:44.826382 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.826149    4282 scope.go:122] "RemoveContainer" containerID="2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc"
Oct 30 21:58:44.826382 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.826077    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:44.834988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.834956    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f47f79-serving-cert\""
Oct 30 21:58:44.851341 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.851316    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"]
Oct 30 21:58:44.851506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.851431    4282 scope.go:122] "RemoveContainer" containerID="2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc"
Oct 30 21:58:44.851980 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:44.851954    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc\": container with ID starting with 2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc not found: ID does not exist" containerID="2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc"
Oct 30 21:58:44.852078 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.851994    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc"} err="failed to get container status \"2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc\": rpc error: code = NotFound desc = could not find container \"2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc\": container with ID starting with 2c02702f330c22f5d8e3f4949d7f48b1ab619a2b20809761fb975af92ffc0ddc not found: ID does not exist"
Oct 30 21:58:44.852078 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.852018    4282 scope.go:122] "RemoveContainer" containerID="2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48"
Oct 30 21:58:44.861880 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.861849    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bf337-658d8d687-trgsf"]
Oct 30 21:58:44.873703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.873675    4282 scope.go:122] "RemoveContainer" containerID="2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48"
Oct 30 21:58:44.874789 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:58:44.874752    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48\": container with ID starting with 2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48 not found: ID does not exist" containerID="2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48"
Oct 30 21:58:44.874948 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.874788    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48"} err="failed to get container status \"2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48\": rpc error: code = NotFound desc = could not find container \"2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48\": container with ID starting with 2863584e8e1498977ab1e29709c1cc36acdd050faad4c67d3a98aa131a2aba48 not found: ID does not exist"
Oct 30 21:58:44.902871 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.902797    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"]
Oct 30 21:58:44.906441 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.906416    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bf337-54997c4787-swm6p"]
Oct 30 21:58:44.925997 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.925962    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jstz9\" (UniqueName: \"kubernetes.io/projected/f49d6420-17c2-4528-8a22-ce65fd9dea28-kube-api-access-jstz9\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:44.926482 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.926436    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-maas-api-tls\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:44.926679 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.926660    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49d6420-17c2-4528-8a22-ce65fd9dea28-openshift-service-ca\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:44.926770 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:44.926738    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-metrics-certs\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.041604 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.030622    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jstz9\" (UniqueName: \"kubernetes.io/projected/f49d6420-17c2-4528-8a22-ce65fd9dea28-kube-api-access-jstz9\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.042144 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.042109    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-maas-api-tls\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.055650 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.055580    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49d6420-17c2-4528-8a22-ce65fd9dea28-openshift-service-ca\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.059854 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.059826    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-metrics-certs\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.080427 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.059704    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-maas-api-tls\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.080702 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.056650    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49d6420-17c2-4528-8a22-ce65fd9dea28-openshift-service-ca\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.080840 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.072163    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jstz9\" (UniqueName: \"kubernetes.io/projected/f49d6420-17c2-4528-8a22-ce65fd9dea28-kube-api-access-jstz9\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.080959 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.080260    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-metrics-certs\") pod \"maas-api-e2e-api-b-f47f79-757748cf4d-frgdf\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.159535 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.159503    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:45.531090 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.530078    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7149abdb-cbb4-45d6-88d3-008adf585246" path="/var/lib/kubelet/pods/7149abdb-cbb4-45d6-88d3-008adf585246/volumes"
Oct 30 21:58:45.531090 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.530983    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991c83df-52d4-4454-b757-27129f8f41c5" path="/var/lib/kubelet/pods/991c83df-52d4-4454-b757-27129f8f41c5/volumes"
Oct 30 21:58:45.636087 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.626260    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"]
Oct 30 21:58:45.953202 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:45.948702    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf" event={"ID":"f49d6420-17c2-4528-8a22-ce65fd9dea28","Type":"ContainerStarted","Data":"874740b4b7b457e8d8c9933efa6800011a92b246c1e5db21f8b93328e9ee37e9"}
Oct 30 21:58:46.705901 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.705871    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"]
Oct 30 21:58:46.711227 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.711200    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:46.714355 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.714314    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3e3670-dockercfg-8tkzn\""
Oct 30 21:58:46.714778 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.714758    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3e3670\""
Oct 30 21:58:46.722094 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.722074    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"]
Oct 30 21:58:46.804552 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.804517    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4\" (UID: \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:46.804742 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.804723    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4h4d\" (UniqueName: \"kubernetes.io/projected/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-kube-api-access-d4h4d\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4\" (UID: \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:46.941504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.935280    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4h4d\" (UniqueName: \"kubernetes.io/projected/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-kube-api-access-d4h4d\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4\" (UID: \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:46.941504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.935624    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4\" (UID: \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:46.941504 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:46.936625    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4\" (UID: \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:47.071493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.057109    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4h4d\" (UniqueName: \"kubernetes.io/projected/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-kube-api-access-d4h4d\") pod \"payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4\" (UID: \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:47.181846 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.181804    4282 generic.go:356] "Generic (PLEG): container finished" podID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerID="f31f67b42b5e2659f32439918c3b0372d6559ae3751f6c992396fa1984774b0c" exitCode=0
Oct 30 21:58:47.182234 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.182210    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" event={"ID":"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd","Type":"ContainerDied","Data":"f31f67b42b5e2659f32439918c3b0372d6559ae3751f6c992396fa1984774b0c"}
Oct 30 21:58:47.182491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.182429    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm" event={"ID":"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd","Type":"ContainerDied","Data":"ef66c2c33008a8451016718f3297a5490d753f371f4f9dec142e653937a1b73b"}
Oct 30 21:58:47.212024 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.211996    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef66c2c33008a8451016718f3297a5490d753f371f4f9dec142e653937a1b73b"
Oct 30 21:58:47.212300 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.212271    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf" event={"ID":"f49d6420-17c2-4528-8a22-ce65fd9dea28","Type":"ContainerStarted","Data":"cce12f3ac92595a818b00e42c8ef96531c9d834766f13315b9da6dc636a96235"}
Oct 30 21:58:47.212501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.212484    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:47.212665 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.182613    4282 scope.go:122] "RemoveContainer" containerID="b7b444e027dc04ac6f5cf34a3d39193f329610f4e1b3f912304327473328b784"
Oct 30 21:58:47.229586 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.223709    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:58:47.260492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.236001    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf" podStartSLOduration=2.725945846 podStartE2EDuration="3.235981084s" totalImagesPullingTime="510.035238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:47.230623974 +0000 UTC m=+1145.888720199" watchObservedRunningTime="2026-10-30 21:58:47.235981084 +0000 UTC m=+1145.894077307"
Oct 30 21:58:47.280096 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.280059    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-token\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-token\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.280516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.280495    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-data\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.280743 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.280727    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istiod-ca-cert\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.280946 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.280930    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-socket\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.281077 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.280765    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-data" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:47.281216 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.281041    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istiod-ca-cert" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:47.281308 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.281085    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-kube-api-access-47z9m\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-kube-api-access-47z9m\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.281986 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.281959    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-socket" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:47.282823 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.282800    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.282941 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.282927    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.283047 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.283031    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.284648 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.284609    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-kube-api-access-47z9m" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "kube-api-access-47z9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:47.287679 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.287657    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-token" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:47.340186 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.340122    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:47.386894 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.385777    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-podinfo\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.386894 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.385872    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-envoy\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.386894 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.386018    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-credential-socket\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.386894 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.386164    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-certs\") pod \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\" (UID: \"29167fb3-7509-4cb7-9ccb-3b6bc345f8dd\") "
Oct 30 21:58:47.388233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.388182    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47z9m\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-kube-api-access-47z9m\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.388233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.388223    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.389187 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.388750    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-certs" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:47.403029 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.402989    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-credential-socket" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:47.441128 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.441016    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-envoy" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:47.441434 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.441398    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-podinfo" pod "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" (UID: "29167fb3-7509-4cb7-9ccb-3b6bc345f8dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:58:47.491100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.490917    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.491100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.490944    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.491100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.490954    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.491100 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.490964    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:47.756420 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.756392    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"]
Oct 30 21:58:47.757660 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:58:47.757631    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c2eb3e_0e10_43af_bc56_c3e00d663ee0.slice/crio-84b6ce01a539eba462b9ae010b923e683efcc06ebb29998131f1f4913ce3de1c WatchSource:0}: Error finding container 84b6ce01a539eba462b9ae010b923e683efcc06ebb29998131f1f4913ce3de1c: Status 404 returned error can't find the container with id 84b6ce01a539eba462b9ae010b923e683efcc06ebb29998131f1f4913ce3de1c
Oct 30 21:58:47.781127 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:47.781102    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:58:48.204501 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.204455    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"
Oct 30 21:58:48.206281 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.206255    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4" event={"ID":"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0","Type":"ContainerStarted","Data":"a93c96e6e6edcfdb5e994a9e76e705695dddf66e38ee22988a40a3643bfbe0cf"}
Oct 30 21:58:48.206410 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.206335    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4" event={"ID":"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0","Type":"ContainerStarted","Data":"84b6ce01a539eba462b9ae010b923e683efcc06ebb29998131f1f4913ce3de1c"}
Oct 30 21:58:48.206410 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.206359    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:48.303708 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.303542    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4" podStartSLOduration=2.30352917 podStartE2EDuration="2.30352917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:48.301516117 +0000 UTC m=+1146.959612342" watchObservedRunningTime="2026-10-30 21:58:48.30352917 +0000 UTC m=+1146.961625392"
Oct 30 21:58:48.364863 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.364817    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"]
Oct 30 21:58:48.374921 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.374890    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-689a2563-openshift-default-d86655557-2ktpm"]
Oct 30 21:58:48.499873 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.499841    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"]
Oct 30 21:58:48.500920 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.500889    4282 cpu_manager.go:414] "containerMap: removing container" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy"
Oct 30 21:58:48.500920 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.500919    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy"
Oct 30 21:58:48.501078 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.500949    4282 cpu_manager.go:414] "containerMap: removing container" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy"
Oct 30 21:58:48.501078 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.500957    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy"
Oct 30 21:58:48.501158 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.501099    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy"
Oct 30 21:58:48.501158 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.501114    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" containerName="istio-proxy"
Oct 30 21:58:48.510780 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.510749    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.517526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.517502    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"]
Oct 30 21:58:48.518864 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.518816    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0b3fe4-serving-cert\""
Oct 30 21:58:48.621194 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.621114    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.621194 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.621174    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.621370 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.621323    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jl86\" (UniqueName: \"kubernetes.io/projected/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-kube-api-access-8jl86\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.621440 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.621419    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-metrics-certs\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.641453 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.641424    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"]
Oct 30 21:58:48.645724 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.645700    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:48.665603 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.665576    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"]
Oct 30 21:58:48.666320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.666306    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8d2c42-dockercfg-7gtcx\""
Oct 30 21:58:48.666603 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.666588    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8d2c42\""
Oct 30 21:58:48.701051 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.701023    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"]
Oct 30 21:58:48.705245 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.705219    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:48.722851 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.722726    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jl86\" (UniqueName: \"kubernetes.io/projected/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-kube-api-access-8jl86\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.722851 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.722740    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"]
Oct 30 21:58:48.722851 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.722828    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-metrics-certs\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.723107 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.723025    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.723107 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.723097    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.724036 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.724011    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.725913 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.725885    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.726735 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.726708    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-metrics-certs\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.743772 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.743739    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jl86\" (UniqueName: \"kubernetes.io/projected/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-kube-api-access-8jl86\") pod \"maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.832494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.827068    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4vkt\" (UniqueName: \"kubernetes.io/projected/179fc693-e7d7-4016-afe4-f45123b21fa9-kube-api-access-c4vkt\") pod \"payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r\" (UID: \"179fc693-e7d7-4016-afe4-f45123b21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:48.832494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.827203    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d27p\" (UniqueName: \"kubernetes.io/projected/c534218e-d003-4a45-82cf-3aafc113fa32-kube-api-access-9d27p\") pod \"payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26\" (UID: \"c534218e-d003-4a45-82cf-3aafc113fa32\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:48.832494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.827254    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179fc693-e7d7-4016-afe4-f45123b21fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r\" (UID: \"179fc693-e7d7-4016-afe4-f45123b21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:48.832494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.827321    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c534218e-d003-4a45-82cf-3aafc113fa32-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26\" (UID: \"c534218e-d003-4a45-82cf-3aafc113fa32\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:48.845496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.843112    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:48.975581 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.944742    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4vkt\" (UniqueName: \"kubernetes.io/projected/179fc693-e7d7-4016-afe4-f45123b21fa9-kube-api-access-c4vkt\") pod \"payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r\" (UID: \"179fc693-e7d7-4016-afe4-f45123b21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:48.975581 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.944893    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d27p\" (UniqueName: \"kubernetes.io/projected/c534218e-d003-4a45-82cf-3aafc113fa32-kube-api-access-9d27p\") pod \"payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26\" (UID: \"c534218e-d003-4a45-82cf-3aafc113fa32\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:48.975581 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.944941    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179fc693-e7d7-4016-afe4-f45123b21fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r\" (UID: \"179fc693-e7d7-4016-afe4-f45123b21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:48.975581 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.945009    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c534218e-d003-4a45-82cf-3aafc113fa32-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26\" (UID: \"c534218e-d003-4a45-82cf-3aafc113fa32\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:48.975581 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.945987    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c534218e-d003-4a45-82cf-3aafc113fa32-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26\" (UID: \"c534218e-d003-4a45-82cf-3aafc113fa32\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:48.975581 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:48.947389    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179fc693-e7d7-4016-afe4-f45123b21fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r\" (UID: \"179fc693-e7d7-4016-afe4-f45123b21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:49.025492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:49.001158    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d27p\" (UniqueName: \"kubernetes.io/projected/c534218e-d003-4a45-82cf-3aafc113fa32-kube-api-access-9d27p\") pod \"payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26\" (UID: \"c534218e-d003-4a45-82cf-3aafc113fa32\") " pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:49.025780 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:49.012022    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4vkt\" (UniqueName: \"kubernetes.io/projected/179fc693-e7d7-4016-afe4-f45123b21fa9-kube-api-access-c4vkt\") pod \"payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r\" (UID: \"179fc693-e7d7-4016-afe4-f45123b21fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:49.261506 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:49.261431    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:49.322961 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:49.322721    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:49.554430 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:49.554397    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29167fb3-7509-4cb7-9ccb-3b6bc345f8dd" path="/var/lib/kubelet/pods/29167fb3-7509-4cb7-9ccb-3b6bc345f8dd/volumes"
Oct 30 21:58:49.555108 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:49.555086    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"]
Oct 30 21:58:49.864816 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:49.850797    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"]
Oct 30 21:58:50.016492 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:49.994625    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"]
Oct 30 21:58:50.224423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.224393    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd" event={"ID":"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e","Type":"ContainerStarted","Data":"ae12cef140086d32aff7264a3773083a82ca57727a619e0548d86b70393ab37d"}
Oct 30 21:58:50.226036 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.226010    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r" event={"ID":"179fc693-e7d7-4016-afe4-f45123b21fa9","Type":"ContainerStarted","Data":"96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69"}
Oct 30 21:58:50.226145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.226063    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r" event={"ID":"179fc693-e7d7-4016-afe4-f45123b21fa9","Type":"ContainerStarted","Data":"6874706286e7a8de467a0ac4033222eb2abf9fb3d5f3688aa35a07e858807f55"}
Oct 30 21:58:50.226145 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.226100    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:58:50.227661 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.227641    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26" event={"ID":"c534218e-d003-4a45-82cf-3aafc113fa32","Type":"ContainerStarted","Data":"b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241"}
Oct 30 21:58:50.227729 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.227676    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26" event={"ID":"c534218e-d003-4a45-82cf-3aafc113fa32","Type":"ContainerStarted","Data":"77cb226433dc7a061635e8d25b1488d1a70f988a10f550c5228874be4ea532d6"}
Oct 30 21:58:50.229167 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.229145    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:58:50.272062 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.271859    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r" podStartSLOduration=2.271835002 podStartE2EDuration="2.271835002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:50.251092787 +0000 UTC m=+1148.909189009" watchObservedRunningTime="2026-10-30 21:58:50.271835002 +0000 UTC m=+1148.929931230"
Oct 30 21:58:50.289647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:50.289602    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26" podStartSLOduration=2.2895876680000002 podStartE2EDuration="2.289587668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:50.288596112 +0000 UTC m=+1148.946692342" watchObservedRunningTime="2026-10-30 21:58:50.289587668 +0000 UTC m=+1148.947683890"
Oct 30 21:58:51.284177 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:51.284137    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd" event={"ID":"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e","Type":"ContainerStarted","Data":"2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f"}
Oct 30 21:58:51.284800 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:51.284778    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:51.306708 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:51.306664    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd" podStartSLOduration=2.689676592 podStartE2EDuration="3.306650928s" totalImagesPullingTime="616.974336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:51.303148051 +0000 UTC m=+1149.961244274" watchObservedRunningTime="2026-10-30 21:58:51.306650928 +0000 UTC m=+1149.964747177"
Oct 30 21:58:51.686524 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:51.686491    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"]
Oct 30 21:58:51.688828 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:51.688796    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp" podUID="7114c471-8f92-4e57-a34a-deda0832dd12" containerName="maas-api" containerID="cri-o://cbc01f4c5f2fe1d00bb6e9147b790e0da365c91f0248acb0899db4b85e527452" gracePeriod=30
Oct 30 21:58:52.392057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.390026    4282 generic.go:356] "Generic (PLEG): container finished" podID="7114c471-8f92-4e57-a34a-deda0832dd12" containerID="cbc01f4c5f2fe1d00bb6e9147b790e0da365c91f0248acb0899db4b85e527452" exitCode=0
Oct 30 21:58:52.392057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.391151    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp" event={"ID":"7114c471-8f92-4e57-a34a-deda0832dd12","Type":"ContainerDied","Data":"cbc01f4c5f2fe1d00bb6e9147b790e0da365c91f0248acb0899db4b85e527452"}
Oct 30 21:58:52.392057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.391206    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp" event={"ID":"7114c471-8f92-4e57-a34a-deda0832dd12","Type":"ContainerDied","Data":"8bc4cf5a4f8e9feb28da6b4adf867cfa117c6cd0c5295e947a7e697597a12ade"}
Oct 30 21:58:52.392057 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.391231    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bc4cf5a4f8e9feb28da6b4adf867cfa117c6cd0c5295e947a7e697597a12ade"
Oct 30 21:58:52.432132 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.432101    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:58:52.548949 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.548921    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-metrics-certs\") pod \"7114c471-8f92-4e57-a34a-deda0832dd12\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") "
Oct 30 21:58:52.549111 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.549059    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-maas-api-tls\") pod \"7114c471-8f92-4e57-a34a-deda0832dd12\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") "
Oct 30 21:58:52.549173 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.549148    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7114c471-8f92-4e57-a34a-deda0832dd12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114c471-8f92-4e57-a34a-deda0832dd12-openshift-service-ca\") pod \"7114c471-8f92-4e57-a34a-deda0832dd12\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") "
Oct 30 21:58:52.549320 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.549215    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7114c471-8f92-4e57-a34a-deda0832dd12-kube-api-access-p26t5\" (UniqueName: \"kubernetes.io/projected/7114c471-8f92-4e57-a34a-deda0832dd12-kube-api-access-p26t5\") pod \"7114c471-8f92-4e57-a34a-deda0832dd12\" (UID: \"7114c471-8f92-4e57-a34a-deda0832dd12\") "
Oct 30 21:58:52.549699 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.549659    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7114c471-8f92-4e57-a34a-deda0832dd12-openshift-service-ca" pod "7114c471-8f92-4e57-a34a-deda0832dd12" (UID: "7114c471-8f92-4e57-a34a-deda0832dd12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:52.550165 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.550124    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7114c471-8f92-4e57-a34a-deda0832dd12-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:52.557486 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.551839    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7114c471-8f92-4e57-a34a-deda0832dd12-kube-api-access-p26t5" pod "7114c471-8f92-4e57-a34a-deda0832dd12" (UID: "7114c471-8f92-4e57-a34a-deda0832dd12"). InnerVolumeSpecName "kube-api-access-p26t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:52.584631 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.584182    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-maas-api-tls" pod "7114c471-8f92-4e57-a34a-deda0832dd12" (UID: "7114c471-8f92-4e57-a34a-deda0832dd12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:52.584631 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.584361    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-metrics-certs" pod "7114c471-8f92-4e57-a34a-deda0832dd12" (UID: "7114c471-8f92-4e57-a34a-deda0832dd12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:52.657495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.652348    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:52.657495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.652386    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7114c471-8f92-4e57-a34a-deda0832dd12-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:52.657495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.652402    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p26t5\" (UniqueName: \"kubernetes.io/projected/7114c471-8f92-4e57-a34a-deda0832dd12-kube-api-access-p26t5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:52.711175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:52.711147    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:58:53.248499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:53.247809    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 21:58:53.406544 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:53.403182    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"
Oct 30 21:58:53.455496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:53.451909    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"]
Oct 30 21:58:53.490669 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:53.490637    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4bf337-b996b7984-98lrp"]
Oct 30 21:58:53.714882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:53.714858    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:58:55.554494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:55.549620    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7114c471-8f92-4e57-a34a-deda0832dd12" path="/var/lib/kubelet/pods/7114c471-8f92-4e57-a34a-deda0832dd12/volumes"
Oct 30 21:58:57.397495 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.397435    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 21:58:57.840619 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.840589    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"]
Oct 30 21:58:57.842373 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.842351    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7114c471-8f92-4e57-a34a-deda0832dd12" containerName="maas-api"
Oct 30 21:58:57.842550 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.842537    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7114c471-8f92-4e57-a34a-deda0832dd12" containerName="maas-api"
Oct 30 21:58:57.842807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.842793    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7114c471-8f92-4e57-a34a-deda0832dd12" containerName="maas-api"
Oct 30 21:58:57.848622 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.848584    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"]
Oct 30 21:58:57.848887 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.848729    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:57.851845 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.851754    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8d2c42-serving-cert\""
Oct 30 21:58:57.942411 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.940516    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-maas-api-tls\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:57.942411 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.940601    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-metrics-certs\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:57.942411 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.940695    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dff73b1-07de-443b-b2e4-077159317931-openshift-service-ca\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:57.942411 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:57.940990    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztmfd\" (UniqueName: \"kubernetes.io/projected/9dff73b1-07de-443b-b2e4-077159317931-kube-api-access-ztmfd\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.046490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.044435    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztmfd\" (UniqueName: \"kubernetes.io/projected/9dff73b1-07de-443b-b2e4-077159317931-kube-api-access-ztmfd\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.046490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.044534    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-maas-api-tls\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.046490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.044586    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-metrics-certs\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.046490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.044671    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dff73b1-07de-443b-b2e4-077159317931-openshift-service-ca\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.059258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.059216    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dff73b1-07de-443b-b2e4-077159317931-openshift-service-ca\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.061807 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.061771    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-maas-api-tls\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.062032 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.062015    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-metrics-certs\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.081418 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.081379    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztmfd\" (UniqueName: \"kubernetes.io/projected/9dff73b1-07de-443b-b2e4-077159317931-kube-api-access-ztmfd\") pod \"maas-api-e2e-api-a-8d2c42-644bf67649-57mt5\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.182670 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.182636    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:58.485908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:58.485881    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"]
Oct 30 21:58:58.488445 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:58:58.488414    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dff73b1_07de_443b_b2e4_077159317931.slice/crio-2f92223dec630997ceddfbcc46c0ee4d467150f32e10371e07a2cfa4903228f6 WatchSource:0}: Error finding container 2f92223dec630997ceddfbcc46c0ee4d467150f32e10371e07a2cfa4903228f6: Status 404 returned error can't find the container with id 2f92223dec630997ceddfbcc46c0ee4d467150f32e10371e07a2cfa4903228f6
Oct 30 21:58:59.221054 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:59.221024    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:58:59.470350 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:59.470323    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5" event={"ID":"9dff73b1-07de-443b-b2e4-077159317931","Type":"ContainerStarted","Data":"38c424312576c23019caad75d8ba1b98313eadc8054dfad1eefc4c1b49576f2d"}
Oct 30 21:58:59.470526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:59.470363    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5" event={"ID":"9dff73b1-07de-443b-b2e4-077159317931","Type":"ContainerStarted","Data":"2f92223dec630997ceddfbcc46c0ee4d467150f32e10371e07a2cfa4903228f6"}
Oct 30 21:58:59.470526 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:59.470514    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:58:59.498458 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:58:59.498409    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5" podStartSLOduration=1.931197501 podStartE2EDuration="2.498396264s" totalImagesPullingTime="567.198763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:59.496148799 +0000 UTC m=+1158.154245022" watchObservedRunningTime="2026-10-30 21:58:59.498396264 +0000 UTC m=+1158.156492487"
Oct 30 21:59:01.306948 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.306910    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:59:01.394321 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.392866    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"]
Oct 30 21:59:01.402685 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.402654    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.421527 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.413071    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c614f9-openshift-default-dockercfg-98lmk\""
Oct 30 21:59:01.421527 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.419828    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"]
Oct 30 21:59:01.528528 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.528494    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-envoy\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.528528 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.528530    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-token\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.528756 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.528648    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-podinfo\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.528756 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.528725    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istiod-ca-cert\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.528882 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.528856    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-credential-socket\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.528944 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.528899    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwf5\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-kube-api-access-prwf5\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.529044 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.529030    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-socket\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.529076 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.529067    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-certs\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.529129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.529084    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-data\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.632919    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-envoy\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.632995    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-token\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.633130    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-podinfo\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.633370    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istiod-ca-cert\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.633485    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-credential-socket\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.633515    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwf5\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-kube-api-access-prwf5\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.633649    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-socket\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.633731    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-certs\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.633755    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-data\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.634519    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-data\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.637542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.636846    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-credential-socket\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.638210 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.637698    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-socket\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.647502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.638393    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-certs\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.647502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.639085    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istiod-ca-cert\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.687161 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.687082    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-podinfo\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.687161 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.687072    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-envoy\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.782516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.781887    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwf5\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-kube-api-access-prwf5\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:01.804594 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:01.803848    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-token\") pod \"e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") " pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:02.027419 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.027240    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:02.393698 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.393672    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"]
Oct 30 21:59:02.395217 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:59:02.395188    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c43be2f_8c29_48e2_b96b_0dec00fa03ff.slice/crio-ef5793d2d2e30e4bfb68080e28a3c25aa1ee3441ada8bfc1850da1f1230060ee WatchSource:0}: Error finding container ef5793d2d2e30e4bfb68080e28a3c25aa1ee3441ada8bfc1850da1f1230060ee: Status 404 returned error can't find the container with id ef5793d2d2e30e4bfb68080e28a3c25aa1ee3441ada8bfc1850da1f1230060ee
Oct 30 21:59:02.395529 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.395508    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:59:02.398025 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.397992    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:02.398114 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.398058    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:02.398114 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.398090    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:02.727806 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.727763    4282 generic.go:356] "Generic (PLEG): container finished" podID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerID="17c232b350d54e703c535f68a4f0c21f828efc255047d1fd6bb4a9d6db547ee1" exitCode=0
Oct 30 21:59:02.727989 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.727837    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" event={"ID":"81fb080c-08ed-4eee-bcd1-893e4c192f69","Type":"ContainerDied","Data":"17c232b350d54e703c535f68a4f0c21f828efc255047d1fd6bb4a9d6db547ee1"}
Oct 30 21:59:02.728316 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.728300    4282 scope.go:122] "RemoveContainer" containerID="17c232b350d54e703c535f68a4f0c21f828efc255047d1fd6bb4a9d6db547ee1"
Oct 30 21:59:02.729964 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.729945    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" event={"ID":"0c43be2f-8c29-48e2-b96b-0dec00fa03ff","Type":"ContainerStarted","Data":"8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795"}
Oct 30 21:59:02.730081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.729980    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" event={"ID":"0c43be2f-8c29-48e2-b96b-0dec00fa03ff","Type":"ContainerStarted","Data":"ef5793d2d2e30e4bfb68080e28a3c25aa1ee3441ada8bfc1850da1f1230060ee"}
Oct 30 21:59:02.730081 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.730027    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:02.730164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.730093    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:02.730164 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.730128    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:02.790333 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:02.786408    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podStartSLOduration=1.786389649 podStartE2EDuration="1.786389649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:02.781335792 +0000 UTC m=+1161.439432041" watchObservedRunningTime="2026-10-30 21:59:02.786389649 +0000 UTC m=+1161.444485872"
Oct 30 21:59:03.034574 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.033800    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:03.036951 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.035200    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"]
Oct 30 21:59:03.072528 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.065913    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:03.079381 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.074028    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"]
Oct 30 21:59:03.782933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.779765    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" event={"ID":"81fb080c-08ed-4eee-bcd1-893e4c192f69","Type":"ContainerStarted","Data":"46064622c183616f56e2444bd35c57343fc92d99dd8fb084d945cd12eb073c35"}
Oct 30 21:59:03.782933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.779858    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:03.782933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.781841    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:03.861914 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.861882    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:59:03.862266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.862245    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:59:03.862417 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:03.862389    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:59:04.081913 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:04.081823    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:59:04.794113 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:59:04.793929    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:59:04.870493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:04.858844    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:59:04.870493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:04.858905    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:59:05.656739 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:05.656712    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 21:59:05.877910 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:05.877871    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:59:05.878547 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:05.878502    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:59:06.874439 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:06.874404    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:59:06.874730 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:06.874700    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:59:07.907496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:07.871304    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:59:07.908144 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:07.908107    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:59:08.040086 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:08.040013    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk" podUID="5bbbb8f8-acdb-4fe8-9fac-88839b560961" containerName="payload-processing" containerID="cri-o://656809f6f3b84bca8bf2097ccf6b0337ff6dc199ed456c1c8335c194ebb4bc5c" gracePeriod=25
Oct 30 21:59:08.077848 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:08.077804    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5" podUID="7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3" containerName="payload-pre-processing" containerID="cri-o://a105b9679b8acc62934d2e016c088c82eea21b79700f1077a131c0516c3d97c9" gracePeriod=25
Oct 30 21:59:08.889973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:08.886207    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:59:08.889973 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:08.886264    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:59:09.024759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.024725    4282 generic.go:356] "Generic (PLEG): container finished" podID="7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3" containerID="a105b9679b8acc62934d2e016c088c82eea21b79700f1077a131c0516c3d97c9" exitCode=0
Oct 30 21:59:09.025138 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.024794    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5" event={"ID":"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3","Type":"ContainerDied","Data":"a105b9679b8acc62934d2e016c088c82eea21b79700f1077a131c0516c3d97c9"}
Oct 30 21:59:09.026865 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.026840    4282 generic.go:356] "Generic (PLEG): container finished" podID="5bbbb8f8-acdb-4fe8-9fac-88839b560961" containerID="656809f6f3b84bca8bf2097ccf6b0337ff6dc199ed456c1c8335c194ebb4bc5c" exitCode=0
Oct 30 21:59:09.026963 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.026949    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk" event={"ID":"5bbbb8f8-acdb-4fe8-9fac-88839b560961","Type":"ContainerDied","Data":"656809f6f3b84bca8bf2097ccf6b0337ff6dc199ed456c1c8335c194ebb4bc5c"}
Oct 30 21:59:09.027150 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.027107    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:59:09.035550 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.033793    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:59:09.144647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.144583    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-kube-api-access-6zp5p\" (UniqueName: \"kubernetes.io/projected/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-kube-api-access-6zp5p\") pod \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\" (UID: \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\") "
Oct 30 21:59:09.144756 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.144669    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbbb8f8-acdb-4fe8-9fac-88839b560961-kube-api-access-bbnvw\" (UniqueName: \"kubernetes.io/projected/5bbbb8f8-acdb-4fe8-9fac-88839b560961-kube-api-access-bbnvw\") pod \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\" (UID: \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\") "
Oct 30 21:59:09.144756 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.144691    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-plugins-config-volume\") pod \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\" (UID: \"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3\") "
Oct 30 21:59:09.144883 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.144871    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbbb8f8-acdb-4fe8-9fac-88839b560961-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbbb8f8-acdb-4fe8-9fac-88839b560961-plugins-config-volume\") pod \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\" (UID: \"5bbbb8f8-acdb-4fe8-9fac-88839b560961\") "
Oct 30 21:59:09.145625 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.145588    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbbb8f8-acdb-4fe8-9fac-88839b560961-plugins-config-volume" pod "5bbbb8f8-acdb-4fe8-9fac-88839b560961" (UID: "5bbbb8f8-acdb-4fe8-9fac-88839b560961"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:09.145871 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.145847    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-plugins-config-volume" pod "7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3" (UID: "7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:09.147923 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.147905    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-kube-api-access-6zp5p" pod "7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3" (UID: "7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3"). InnerVolumeSpecName "kube-api-access-6zp5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:09.148295 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.148275    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbbb8f8-acdb-4fe8-9fac-88839b560961-kube-api-access-bbnvw" pod "5bbbb8f8-acdb-4fe8-9fac-88839b560961" (UID: "5bbbb8f8-acdb-4fe8-9fac-88839b560961"). InnerVolumeSpecName "kube-api-access-bbnvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:09.246708 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.246665    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bbbb8f8-acdb-4fe8-9fac-88839b560961-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:09.246708 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.246705    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zp5p\" (UniqueName: \"kubernetes.io/projected/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-kube-api-access-6zp5p\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:09.246708 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.246715    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbnvw\" (UniqueName: \"kubernetes.io/projected/5bbbb8f8-acdb-4fe8-9fac-88839b560961-kube-api-access-bbnvw\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:09.246951 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.246745    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:09.869540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.869043    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:59:09.869540 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:09.869302    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:59:10.035062 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.035034    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"
Oct 30 21:59:10.035583 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.035081    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk" event={"ID":"5bbbb8f8-acdb-4fe8-9fac-88839b560961","Type":"ContainerDied","Data":"caab1adff54a9fa6868a10883eeffe1ebca68385e3fcd0b413fda2b46efe2705"}
Oct 30 21:59:10.035583 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.035147    4282 scope.go:122] "RemoveContainer" containerID="656809f6f3b84bca8bf2097ccf6b0337ff6dc199ed456c1c8335c194ebb4bc5c"
Oct 30 21:59:10.037633 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.037610    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5" event={"ID":"7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3","Type":"ContainerDied","Data":"7452742d4bf880c2ab2de8bc45953e233fadb1ff7e3b9288af009a1ad40e91b2"}
Oct 30 21:59:10.037749 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.037702    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"
Oct 30 21:59:10.047791 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.047774    4282 scope.go:122] "RemoveContainer" containerID="a105b9679b8acc62934d2e016c088c82eea21b79700f1077a131c0516c3d97c9"
Oct 30 21:59:10.107610 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.107580    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"]
Oct 30 21:59:10.111332 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.111305    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0b3fe4-6f6c97c49-bh6jk"]
Oct 30 21:59:10.132204 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.131400    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"]
Oct 30 21:59:10.135743 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.135722    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0b3fe4-85d467cf95-6cmd5"]
Oct 30 21:59:10.870490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.868068    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Oct 30 21:59:10.870490 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:10.868127    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Oct 30 21:59:11.113859 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:11.113827    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"]
Oct 30 21:59:11.541494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:11.537874    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbbb8f8-acdb-4fe8-9fac-88839b560961" path="/var/lib/kubelet/pods/5bbbb8f8-acdb-4fe8-9fac-88839b560961/volumes"
Oct 30 21:59:11.541494 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:11.538560    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3" path="/var/lib/kubelet/pods/7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3/volumes"
Oct 30 21:59:11.896511 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:11.895929    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:59:11.897499 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:11.897410    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 21:59:13.664493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.660124    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"]
Oct 30 21:59:13.664493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.661015    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3" containerName="payload-pre-processing"
Oct 30 21:59:13.664493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.661033    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3" containerName="payload-pre-processing"
Oct 30 21:59:13.664493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.661065    4282 cpu_manager.go:414] "containerMap: removing container" podUID="5bbbb8f8-acdb-4fe8-9fac-88839b560961" containerName="payload-processing"
Oct 30 21:59:13.664493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.661072    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbbb8f8-acdb-4fe8-9fac-88839b560961" containerName="payload-processing"
Oct 30 21:59:13.664493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.661243    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1fd603-5d60-4e66-bd3b-1d67f1b34fc3" containerName="payload-pre-processing"
Oct 30 21:59:13.664493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.661258    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbbb8f8-acdb-4fe8-9fac-88839b560961" containerName="payload-processing"
Oct 30 21:59:13.936699 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.936663    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"]
Oct 30 21:59:13.937045 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.937025    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:13.966941 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:13.966917    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8c6cf1-openshift-default-dockercfg-ztq6w\""
Oct 30 21:59:14.028053 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028005    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-data\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.028222 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028159    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-certs\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.028266 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028236    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-socket\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.028301 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028280    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-istio-token\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.028339 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028304    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc999841-a85f-49d0-942e-7a09f76df056-istiod-ca-cert\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.028408 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028391    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-credential-socket\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.028443 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028419    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmh8\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-kube-api-access-pkmh8\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.028513 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028451    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-envoy\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.028593 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.028580    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc999841-a85f-49d0-942e-7a09f76df056-istio-podinfo\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.132679    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc999841-a85f-49d0-942e-7a09f76df056-istio-podinfo\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.132876    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-data\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.133006    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-certs\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.133094    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-socket\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.133150    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-istio-token\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.133178    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc999841-a85f-49d0-942e-7a09f76df056-istiod-ca-cert\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.133236    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-credential-socket\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.133264    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmh8\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-kube-api-access-pkmh8\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.134493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.133313    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-envoy\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.160925 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.160891    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-socket\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.183493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.163808    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-data\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.183493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.164092    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-certs\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.183493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.164782    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc999841-a85f-49d0-942e-7a09f76df056-istiod-ca-cert\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.183493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.165440    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-credential-socket\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.183493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.177830    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-envoy\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.192859 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.192784    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc999841-a85f-49d0-942e-7a09f76df056-istio-podinfo\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.202169 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.202142    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmh8\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-kube-api-access-pkmh8\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.202688 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.202668    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-istio-token\") pod \"e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") " pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.286819 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.286783    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:14.673052 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.672806    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"]
Oct 30 21:59:14.683805 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:59:14.683776    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc999841_a85f_49d0_942e_7a09f76df056.slice/crio-e7e3597f7f6d658934c1d568746b6a5201130cb0d405334bb4d47b5ced345c72 WatchSource:0}: Error finding container e7e3597f7f6d658934c1d568746b6a5201130cb0d405334bb4d47b5ced345c72: Status 404 returned error can't find the container with id e7e3597f7f6d658934c1d568746b6a5201130cb0d405334bb4d47b5ced345c72
Oct 30 21:59:14.687110 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.687069    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:14.687227 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.687163    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:14.687227 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:14.687207    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:15.301539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:15.300165    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" event={"ID":"dc999841-a85f-49d0-942e-7a09f76df056","Type":"ContainerStarted","Data":"d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528"}
Oct 30 21:59:15.301539 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:15.300240    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" event={"ID":"dc999841-a85f-49d0-942e-7a09f76df056","Type":"ContainerStarted","Data":"e7e3597f7f6d658934c1d568746b6a5201130cb0d405334bb4d47b5ced345c72"}
Oct 30 21:59:15.347496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:15.347429    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podStartSLOduration=2.347413978 podStartE2EDuration="2.347413978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:15.344102223 +0000 UTC m=+1174.002198447" watchObservedRunningTime="2026-10-30 21:59:15.347413978 +0000 UTC m=+1174.005510201"
Oct 30 21:59:16.114522 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.114460    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4" podUID="b9c2eb3e-0e10-43af-bc56-c3e00d663ee0" containerName="payload-pre-processing" containerID="cri-o://a93c96e6e6edcfdb5e994a9e76e705695dddf66e38ee22988a40a3643bfbe0cf" gracePeriod=25
Oct 30 21:59:16.287516 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.287391    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:16.294867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.294539    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:16.311146 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.311116    4282 generic.go:356] "Generic (PLEG): container finished" podID="b9c2eb3e-0e10-43af-bc56-c3e00d663ee0" containerID="a93c96e6e6edcfdb5e994a9e76e705695dddf66e38ee22988a40a3643bfbe0cf" exitCode=0
Oct 30 21:59:16.311330 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.311189    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4" event={"ID":"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0","Type":"ContainerDied","Data":"a93c96e6e6edcfdb5e994a9e76e705695dddf66e38ee22988a40a3643bfbe0cf"}
Oct 30 21:59:16.311919 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.311635    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:16.312996 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.312977    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 21:59:16.615434 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.615407    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:59:16.708637 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.708608    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-plugins-config-volume\") pod \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\" (UID: \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\") "
Oct 30 21:59:16.708837 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.708778    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-kube-api-access-d4h4d\" (UniqueName: \"kubernetes.io/projected/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-kube-api-access-d4h4d\") pod \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\" (UID: \"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0\") "
Oct 30 21:59:16.709362 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.709309    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-plugins-config-volume" pod "b9c2eb3e-0e10-43af-bc56-c3e00d663ee0" (UID: "b9c2eb3e-0e10-43af-bc56-c3e00d663ee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:16.712698 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.711448    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-kube-api-access-d4h4d" pod "b9c2eb3e-0e10-43af-bc56-c3e00d663ee0" (UID: "b9c2eb3e-0e10-43af-bc56-c3e00d663ee0"). InnerVolumeSpecName "kube-api-access-d4h4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:16.820541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.813868    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4h4d\" (UniqueName: \"kubernetes.io/projected/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-kube-api-access-d4h4d\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:16.820541 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:16.813910    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:17.319122 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:17.319095    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"
Oct 30 21:59:17.319588 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:17.319129    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4" event={"ID":"b9c2eb3e-0e10-43af-bc56-c3e00d663ee0","Type":"ContainerDied","Data":"84b6ce01a539eba462b9ae010b923e683efcc06ebb29998131f1f4913ce3de1c"}
Oct 30 21:59:17.319588 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:17.319196    4282 scope.go:122] "RemoveContainer" containerID="a93c96e6e6edcfdb5e994a9e76e705695dddf66e38ee22988a40a3643bfbe0cf"
Oct 30 21:59:17.359222 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:17.359194    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"]
Oct 30 21:59:17.382269 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:17.382241    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3e3670-b8c98b467-gpxp4"]
Oct 30 21:59:17.480983 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:17.480943    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c2eb3e-0e10-43af-bc56-c3e00d663ee0" path="/var/lib/kubelet/pods/b9c2eb3e-0e10-43af-bc56-c3e00d663ee0/volumes"
Oct 30 21:59:19.020484 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.020434    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"]
Oct 30 21:59:19.021101 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.021087    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b9c2eb3e-0e10-43af-bc56-c3e00d663ee0" containerName="payload-pre-processing"
Oct 30 21:59:19.021155 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.021102    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c2eb3e-0e10-43af-bc56-c3e00d663ee0" containerName="payload-pre-processing"
Oct 30 21:59:19.021237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.021227    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c2eb3e-0e10-43af-bc56-c3e00d663ee0" containerName="payload-pre-processing"
Oct 30 21:59:19.024791 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.024773    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:19.034742 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.034716    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8c6cf1-dockercfg-swl7d\""
Oct 30 21:59:19.034886 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.034727    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8c6cf1\""
Oct 30 21:59:19.036930 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.036908    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"]
Oct 30 21:59:19.170258 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.170217    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce10b9-1687-4796-aa7e-8479ea1e6dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn\" (UID: \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:19.170525 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.170455    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbn49\" (UniqueName: \"kubernetes.io/projected/51ce10b9-1687-4796-aa7e-8479ea1e6dde-kube-api-access-hbn49\") pod \"payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn\" (UID: \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:19.272058 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.272019    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbn49\" (UniqueName: \"kubernetes.io/projected/51ce10b9-1687-4796-aa7e-8479ea1e6dde-kube-api-access-hbn49\") pod \"payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn\" (UID: \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:19.272339 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.272316    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce10b9-1687-4796-aa7e-8479ea1e6dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn\" (UID: \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:19.273205 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.273181    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce10b9-1687-4796-aa7e-8479ea1e6dde-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn\" (UID: \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:19.284930 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.284905    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbn49\" (UniqueName: \"kubernetes.io/projected/51ce10b9-1687-4796-aa7e-8479ea1e6dde-kube-api-access-hbn49\") pod \"payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn\" (UID: \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:19.337913 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.337886    4282 generic.go:356] "Generic (PLEG): container finished" podID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerID="b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18" exitCode=0
Oct 30 21:59:19.338079 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.337963    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" event={"ID":"68171810-3eb8-4c3a-a758-f11eaafe5082","Type":"ContainerDied","Data":"b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18"}
Oct 30 21:59:19.338462 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.338446    4282 scope.go:122] "RemoveContainer" containerID="b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18"
Oct 30 21:59:19.340171 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.340143    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:19.340238 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.340214    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:19.340278 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.340245    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:19.341641 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.341625    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:19.419491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:19.412562    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:59:20.065129 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.065098    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"]
Oct 30 21:59:20.066221 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:59:20.066183    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ce10b9_1687_4796_aa7e_8479ea1e6dde.slice/crio-61a64c89800298317115da813b5110ac0f81b5ce3dd14d2acce656b0d9429e45 WatchSource:0}: Error finding container 61a64c89800298317115da813b5110ac0f81b5ce3dd14d2acce656b0d9429e45: Status 404 returned error can't find the container with id 61a64c89800298317115da813b5110ac0f81b5ce3dd14d2acce656b0d9429e45
Oct 30 21:59:20.351558 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.351398    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" event={"ID":"68171810-3eb8-4c3a-a758-f11eaafe5082","Type":"ContainerStarted","Data":"cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce"}
Oct 30 21:59:20.355491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.354714    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:20.355491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.354800    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn" event={"ID":"51ce10b9-1687-4796-aa7e-8479ea1e6dde","Type":"ContainerStarted","Data":"8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e"}
Oct 30 21:59:20.355491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.354856    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn" event={"ID":"51ce10b9-1687-4796-aa7e-8479ea1e6dde","Type":"ContainerStarted","Data":"61a64c89800298317115da813b5110ac0f81b5ce3dd14d2acce656b0d9429e45"}
Oct 30 21:59:20.377491 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.374375    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn" podStartSLOduration=1.374355014 podStartE2EDuration="1.374355014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:20.371903299 +0000 UTC m=+1179.029999535" watchObservedRunningTime="2026-10-30 21:59:20.374355014 +0000 UTC m=+1179.032451239"
Oct 30 21:59:20.606545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.604522    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:59:20.606545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.606355    4282 patch_prober.go:29] interesting pod/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:59:20.606545 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:20.606421    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:59:21.618497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:21.613898    4282 patch_prober.go:29] interesting pod/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:59:21.618497 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:21.613973    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:59:22.604423 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:22.604395    4282 patch_prober.go:29] interesting pod/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:59:22.604641 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:22.604446    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:59:23.604651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:23.604615    4282 patch_prober.go:29] interesting pod/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:59:23.605058 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:23.604669    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:59:24.633640 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:24.633600    4282 patch_prober.go:29] interesting pod/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:59:24.634560 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:24.634517    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:59:25.604838 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:25.604808    4282 patch_prober.go:29] interesting pod/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 21:59:25.605033 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:25.604867    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 21:59:26.609453 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:26.609418    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:59:26.609870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:26.609719    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:59:26.610502 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:26.610460    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 21:59:27.464647 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.456117    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"]
Oct 30 21:59:27.762566 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.750875    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"]
Oct 30 21:59:27.762566 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.756664    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.777644 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.777617    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8c6cf1-serving-cert\""
Oct 30 21:59:27.783522 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.783485    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwzhh\" (UniqueName: \"kubernetes.io/projected/f879727f-02e4-4126-af3d-34b4bc97279a-kube-api-access-kwzhh\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.784089 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.784060    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-maas-api-tls\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.787154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.785545    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-metrics-certs\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.787154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.785583    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f879727f-02e4-4126-af3d-34b4bc97279a-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.933527 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.904933    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-metrics-certs\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.933812 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.933787    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f879727f-02e4-4126-af3d-34b4bc97279a-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.934176 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.934155    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwzhh\" (UniqueName: \"kubernetes.io/projected/f879727f-02e4-4126-af3d-34b4bc97279a-kube-api-access-kwzhh\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.934449 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.934398    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-maas-api-tls\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.935891 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.935868    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f879727f-02e4-4126-af3d-34b4bc97279a-openshift-service-ca\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:27.969741 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.969710    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-maas-api-tls\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:28.015115 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:27.995934    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwzhh\" (UniqueName: \"kubernetes.io/projected/f879727f-02e4-4126-af3d-34b4bc97279a-kube-api-access-kwzhh\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:28.025169 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:28.025128    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-metrics-certs\") pod \"maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:28.129253 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:28.129217    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:28.878685 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:28.878654    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"]
Oct 30 21:59:29.845948 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:29.845900    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn" event={"ID":"f879727f-02e4-4126-af3d-34b4bc97279a","Type":"ContainerStarted","Data":"0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a"}
Oct 30 21:59:29.846155 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:29.845973    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn" event={"ID":"f879727f-02e4-4126-af3d-34b4bc97279a","Type":"ContainerStarted","Data":"03ca674da047467e4ba3da037f6092931d0ed4516abcea236f9d6c463a27c351"}
Oct 30 21:59:29.847330 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:29.847304    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:29.871091 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:29.871041    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn" podStartSLOduration=2.4546397620000002 podStartE2EDuration="2.871026129s" totalImagesPullingTime="416.386367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:29.868845137 +0000 UTC m=+1188.526941360" watchObservedRunningTime="2026-10-30 21:59:29.871026129 +0000 UTC m=+1188.529122352"
Oct 30 21:59:30.739874 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.739841    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"]
Oct 30 21:59:30.744486 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.744445    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.763063 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.763040    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-19cab1-openshift-default-dockercfg-pcrqq\""
Oct 30 21:59:30.771335 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.771309    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"]
Oct 30 21:59:30.823639 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.823606    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7646623d-2c6a-4b60-90f8-31b67cb69c05-istiod-ca-cert\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.823787 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.823650    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzbt\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-kube-api-access-2gzbt\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.823787 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.823721    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-token\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.823890 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.823785    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-certs\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.824049 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.824029    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-credential-socket\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.824116 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.824090    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-envoy\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.824152 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.824133    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-data\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.824188 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.824170    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-socket\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.824232 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.824202    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-podinfo\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925507 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925457    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-credential-socket\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925656 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925552    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-envoy\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925656 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925585    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-data\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925656 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925629    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-socket\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925802 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925679    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-podinfo\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925802 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925777    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7646623d-2c6a-4b60-90f8-31b67cb69c05-istiod-ca-cert\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925808    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzbt\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-kube-api-access-2gzbt\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925862    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-token\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.925908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925892    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-certs\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.926063 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925924    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-credential-socket\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.926063 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.925999    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-data\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.926063 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.926036    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-socket\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.926279 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.926259    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-certs\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.927058 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.927030    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7646623d-2c6a-4b60-90f8-31b67cb69c05-istiod-ca-cert\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.928237 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.928214    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-envoy\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.928756 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.928732    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-podinfo\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.937422 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.937397    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzbt\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-kube-api-access-2gzbt\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:30.937922 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:30.937904    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-token\") pod \"e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") " pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:31.078233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:31.076880    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:31.414542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:31.406701    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 21:59:31.673651 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:31.673629    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"]
Oct 30 21:59:31.674704 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:59:31.674680    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7646623d_2c6a_4b60_90f8_31b67cb69c05.slice/crio-833621a4476649d516e7b10c1247498cad225679883ce6b652abc707f316ffcc WatchSource:0}: Error finding container 833621a4476649d516e7b10c1247498cad225679883ce6b652abc707f316ffcc: Status 404 returned error can't find the container with id 833621a4476649d516e7b10c1247498cad225679883ce6b652abc707f316ffcc
Oct 30 21:59:31.677996 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:31.677957    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:31.678096 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:31.678046    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:31.678096 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:31.678080    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:31.935239 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:31.935094    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" event={"ID":"7646623d-2c6a-4b60-90f8-31b67cb69c05","Type":"ContainerStarted","Data":"833621a4476649d516e7b10c1247498cad225679883ce6b652abc707f316ffcc"}
Oct 30 21:59:32.570360 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:32.570328    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 21:59:33.128703 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:33.128655    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" event={"ID":"7646623d-2c6a-4b60-90f8-31b67cb69c05","Type":"ContainerStarted","Data":"cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e"}
Oct 30 21:59:33.159227 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:33.159178    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podStartSLOduration=3.159161554 podStartE2EDuration="3.159161554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:33.153005831 +0000 UTC m=+1191.811102057" watchObservedRunningTime="2026-10-30 21:59:33.159161554 +0000 UTC m=+1191.817257777"
Oct 30 21:59:34.082870 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:34.082837    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:34.247391 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:34.247358    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:34.649852 ip-10-0-131-245 kubenswrapper[4282]: E1030 21:59:34.649814    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 21:59:35.167046 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:35.167018    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:35.168016 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:35.167998    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 21:59:36.941505 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:36.941440    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 21:59:43.511233 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.510600    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"]
Oct 30 21:59:43.527021 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.526993    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"]
Oct 30 21:59:43.527171 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.527104    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.535621 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.535453    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3654d5\"/\"ipp-route-3654d5-kserve-self-signed-certs\""
Oct 30 21:59:43.535621 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.535563    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3654d5\"/\"default-dockercfg-sgfxm\""
Oct 30 21:59:43.535955 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.535929    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3654d5\"/\"openshift-service-ca.crt\""
Oct 30 21:59:43.536069 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.536006    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3654d5\"/\"kube-root-ca.crt\""
Oct 30 21:59:43.546605 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.542932    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-dshm\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.546605 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.543044    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tmp-dir\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.546605 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.543090    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-model-cache\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.546605 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.543125    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tls-certs\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.546605 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.543360    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-home\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.546605 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.543458    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6p4\" (UniqueName: \"kubernetes.io/projected/7929d5a4-5bdb-4fac-82a0-514d24b190c6-kube-api-access-cl6p4\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.685554 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.645676    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-dshm\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.727819 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.727775    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tmp-dir\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.728118 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.728091    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-model-cache\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.728267 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.728250    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tls-certs\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.728745 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.728706    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-home\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.729025 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.729001    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6p4\" (UniqueName: \"kubernetes.io/projected/7929d5a4-5bdb-4fac-82a0-514d24b190c6-kube-api-access-cl6p4\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.729643 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.714877    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-dshm\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.733856 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.733833    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-home\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.737564 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.732069    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tmp-dir\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:43.737699 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:43.731849    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-model-cache\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:44.035542 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:44.032533    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tls-certs\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:44.100165 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:44.100125    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6p4\" (UniqueName: \"kubernetes.io/projected/7929d5a4-5bdb-4fac-82a0-514d24b190c6-kube-api-access-cl6p4\") pod \"ipp-route-3654d5-kserve-679c964f77-l5fvx\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") " pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:44.151407 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:44.151368    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:44.505947 ip-10-0-131-245 kubenswrapper[4282]: W1030 21:59:44.505910    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7929d5a4_5bdb_4fac_82a0_514d24b190c6.slice/crio-388bf15072402c1cf825a1c5105c8fdcb6bfc6ecc33a87350c210055371d8e1f WatchSource:0}: Error finding container 388bf15072402c1cf825a1c5105c8fdcb6bfc6ecc33a87350c210055371d8e1f: Status 404 returned error can't find the container with id 388bf15072402c1cf825a1c5105c8fdcb6bfc6ecc33a87350c210055371d8e1f
Oct 30 21:59:44.507967 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:44.507941    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"]
Oct 30 21:59:44.613824 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:44.613783    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx" event={"ID":"7929d5a4-5bdb-4fac-82a0-514d24b190c6","Type":"ContainerStarted","Data":"01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975"}
Oct 30 21:59:44.614204 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:44.613845    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx" event={"ID":"7929d5a4-5bdb-4fac-82a0-514d24b190c6","Type":"ContainerStarted","Data":"388bf15072402c1cf825a1c5105c8fdcb6bfc6ecc33a87350c210055371d8e1f"}
Oct 30 21:59:44.633793 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:44.633741    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx" podStartSLOduration=1.63372712 podStartE2EDuration="1.63372712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:44.631221753 +0000 UTC m=+1203.289317976" watchObservedRunningTime="2026-10-30 21:59:44.63372712 +0000 UTC m=+1203.291823342"
Oct 30 21:59:45.175488 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:45.168577    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:45.841618 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:45.841570    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:59:45.842131 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:45.841622    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:49.668079 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:49.668042    4282 generic.go:356] "Generic (PLEG): container finished" podID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerID="e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548" exitCode=0
Oct 30 21:59:49.668638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:49.668097    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" event={"ID":"aa4a5ed4-af4c-4c26-8600-bae241eb1982","Type":"ContainerDied","Data":"e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548"}
Oct 30 21:59:49.668638 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:49.668617    4282 scope.go:122] "RemoveContainer" containerID="e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548"
Oct 30 21:59:49.670443 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:49.670400    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:49.670592 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:49.670503    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:49.670592 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:49.670546    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:49.821709 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:49.821654    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:59:49.821905 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:49.821722    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:50.766493 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:50.756105    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" event={"ID":"aa4a5ed4-af4c-4c26-8600-bae241eb1982","Type":"ContainerStarted","Data":"e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc"}
Oct 30 21:59:51.234496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:51.234450    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:59:51.234725 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:51.234704    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:59:51.234783 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:51.234755    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 21:59:52.237165 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:52.237103    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:59:52.237867 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:52.237834    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 21:59:53.235392 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:53.235158    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:59:53.235819 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:53.235757    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 21:59:54.107607 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:54.107533    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"]
Oct 30 21:59:54.110068 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:54.110041    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"]
Oct 30 21:59:54.220194 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:54.220159    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:54.234809 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:54.234772    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:59:54.234996 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:54.234846    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 21:59:54.241134 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:54.241102    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:55.241964 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:55.237413    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:59:55.241964 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:55.237500    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 21:59:55.327265 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:55.326972    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 21:59:56.237580 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:56.237547    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:59:56.237908 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:56.237880    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 30 21:59:56.948602 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:56.948572    4282 generic.go:356] "Generic (PLEG): container finished" podID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerID="8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795" exitCode=0
Oct 30 21:59:56.948982 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:56.948656    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" event={"ID":"0c43be2f-8c29-48e2-b96b-0dec00fa03ff","Type":"ContainerDied","Data":"8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795"}
Oct 30 21:59:56.949202 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:56.949185    4282 scope.go:122] "RemoveContainer" containerID="8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795"
Oct 30 21:59:56.950820 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:56.950785    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:56.950933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:56.950875    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:56.950933 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:56.950924    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 21:59:57.239603 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:57.239524    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:59:57.240126 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:57.240101    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:59:57.240930 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:57.240908    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 21:59:57.981496 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:57.981291    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" event={"ID":"0c43be2f-8c29-48e2-b96b-0dec00fa03ff","Type":"ContainerStarted","Data":"d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb"}
Oct 30 21:59:58.033007 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:58.032972    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 21:59:58.033188 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:58.033170    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 21:59:58.033273 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:58.033251    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 21:59:59.033455 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.033419    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 21:59:59.033977 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.033502    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 21:59:59.109097 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.109054    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26" podUID="c534218e-d003-4a45-82cf-3aafc113fa32" containerName="payload-processing" containerID="cri-o://b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241" gracePeriod=25
Oct 30 21:59:59.110627 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.110564    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r" podUID="179fc693-e7d7-4016-afe4-f45123b21fa9" containerName="payload-pre-processing" containerID="cri-o://96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69" gracePeriod=25
Oct 30 21:59:59.598988 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.598964    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:59:59.729220 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.729183    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:59:59.733201 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.733180    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c534218e-d003-4a45-82cf-3aafc113fa32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c534218e-d003-4a45-82cf-3aafc113fa32-plugins-config-volume\") pod \"c534218e-d003-4a45-82cf-3aafc113fa32\" (UID: \"c534218e-d003-4a45-82cf-3aafc113fa32\") "
Oct 30 21:59:59.733379 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.733365    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c534218e-d003-4a45-82cf-3aafc113fa32-kube-api-access-9d27p\" (UniqueName: \"kubernetes.io/projected/c534218e-d003-4a45-82cf-3aafc113fa32-kube-api-access-9d27p\") pod \"c534218e-d003-4a45-82cf-3aafc113fa32\" (UID: \"c534218e-d003-4a45-82cf-3aafc113fa32\") "
Oct 30 21:59:59.733620 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.733595    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c534218e-d003-4a45-82cf-3aafc113fa32-plugins-config-volume" pod "c534218e-d003-4a45-82cf-3aafc113fa32" (UID: "c534218e-d003-4a45-82cf-3aafc113fa32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:59.734175 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.734153    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c534218e-d003-4a45-82cf-3aafc113fa32-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:59.735967 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.735944    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c534218e-d003-4a45-82cf-3aafc113fa32-kube-api-access-9d27p" pod "c534218e-d003-4a45-82cf-3aafc113fa32" (UID: "c534218e-d003-4a45-82cf-3aafc113fa32"). InnerVolumeSpecName "kube-api-access-9d27p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:59.835603 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.835528    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179fc693-e7d7-4016-afe4-f45123b21fa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179fc693-e7d7-4016-afe4-f45123b21fa9-plugins-config-volume\") pod \"179fc693-e7d7-4016-afe4-f45123b21fa9\" (UID: \"179fc693-e7d7-4016-afe4-f45123b21fa9\") "
Oct 30 21:59:59.835759 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.835654    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179fc693-e7d7-4016-afe4-f45123b21fa9-kube-api-access-c4vkt\" (UniqueName: \"kubernetes.io/projected/179fc693-e7d7-4016-afe4-f45123b21fa9-kube-api-access-c4vkt\") pod \"179fc693-e7d7-4016-afe4-f45123b21fa9\" (UID: \"179fc693-e7d7-4016-afe4-f45123b21fa9\") "
Oct 30 21:59:59.835958 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.835931    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179fc693-e7d7-4016-afe4-f45123b21fa9-plugins-config-volume" pod "179fc693-e7d7-4016-afe4-f45123b21fa9" (UID: "179fc693-e7d7-4016-afe4-f45123b21fa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:59.836661 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.836638    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/179fc693-e7d7-4016-afe4-f45123b21fa9-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:59.836707 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.836669    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d27p\" (UniqueName: \"kubernetes.io/projected/c534218e-d003-4a45-82cf-3aafc113fa32-kube-api-access-9d27p\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:59.838044 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.838022    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179fc693-e7d7-4016-afe4-f45123b21fa9-kube-api-access-c4vkt" pod "179fc693-e7d7-4016-afe4-f45123b21fa9" (UID: "179fc693-e7d7-4016-afe4-f45123b21fa9"). InnerVolumeSpecName "kube-api-access-c4vkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:59.938959 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.938921    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4vkt\" (UniqueName: \"kubernetes.io/projected/179fc693-e7d7-4016-afe4-f45123b21fa9-kube-api-access-c4vkt\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:59.997642 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.997606    4282 generic.go:356] "Generic (PLEG): container finished" podID="179fc693-e7d7-4016-afe4-f45123b21fa9" containerID="96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69" exitCode=0
Oct 30 21:59:59.997825 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.997675    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"
Oct 30 21:59:59.997825 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.997696    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r" event={"ID":"179fc693-e7d7-4016-afe4-f45123b21fa9","Type":"ContainerDied","Data":"96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69"}
Oct 30 21:59:59.997825 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.997752    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r" event={"ID":"179fc693-e7d7-4016-afe4-f45123b21fa9","Type":"ContainerDied","Data":"6874706286e7a8de467a0ac4033222eb2abf9fb3d5f3688aa35a07e858807f55"}
Oct 30 21:59:59.997825 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.997780    4282 scope.go:122] "RemoveContainer" containerID="96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69"
Oct 30 21:59:59.999154 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.999128    4282 generic.go:356] "Generic (PLEG): container finished" podID="c534218e-d003-4a45-82cf-3aafc113fa32" containerID="b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241" exitCode=0
Oct 30 21:59:59.999260 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.999172    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26" event={"ID":"c534218e-d003-4a45-82cf-3aafc113fa32","Type":"ContainerDied","Data":"b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241"}
Oct 30 21:59:59.999260 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.999186    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"
Oct 30 21:59:59.999260 ip-10-0-131-245 kubenswrapper[4282]: I1030 21:59:59.999211    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26" event={"ID":"c534218e-d003-4a45-82cf-3aafc113fa32","Type":"ContainerDied","Data":"77cb226433dc7a061635e8d25b1488d1a70f988a10f550c5228874be4ea532d6"}
Oct 30 22:00:00.010305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.010272    4282 scope.go:122] "RemoveContainer" containerID="96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69"
Oct 30 22:00:00.010693 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:00.010669    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69\": container with ID starting with 96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69 not found: ID does not exist" containerID="96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69"
Oct 30 22:00:00.010810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.010702    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69"} err="failed to get container status \"96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69\": rpc error: code = NotFound desc = could not find container \"96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69\": container with ID starting with 96963fcc72ed01fae14909a071d15e786cde86bc1e85d31e6cd94c2ee465bf69 not found: ID does not exist"
Oct 30 22:00:00.010810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.010721    4282 scope.go:122] "RemoveContainer" containerID="b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241"
Oct 30 22:00:00.023739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.023715    4282 scope.go:122] "RemoveContainer" containerID="b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241"
Oct 30 22:00:00.024115 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:00.024095    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241\": container with ID starting with b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241 not found: ID does not exist" containerID="b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241"
Oct 30 22:00:00.024176 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.024127    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241"} err="failed to get container status \"b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241\": rpc error: code = NotFound desc = could not find container \"b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241\": container with ID starting with b0f4e990d2241e85598beb30b8c251614ed0bbd752f0c1fc2326088589615241 not found: ID does not exist"
Oct 30 22:00:00.030768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.030728    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"]
Oct 30 22:00:00.034609 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.034578    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:00:00.035108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.034639    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 22:00:00.041278 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.041248    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8d2c42-55989d8d79-xtf7r"]
Oct 30 22:00:00.088213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.088133    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"]
Oct 30 22:00:00.093126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.093098    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8d2c42-7d969d8f65-56l26"]
Oct 30 22:00:00.419607 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.419572    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"]
Oct 30 22:00:00.420579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.420447    4282 cpu_manager.go:414] "containerMap: removing container" podUID="179fc693-e7d7-4016-afe4-f45123b21fa9" containerName="payload-pre-processing"
Oct 30 22:00:00.420579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.420515    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179fc693-e7d7-4016-afe4-f45123b21fa9" containerName="payload-pre-processing"
Oct 30 22:00:00.420579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.420536    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c534218e-d003-4a45-82cf-3aafc113fa32" containerName="payload-processing"
Oct 30 22:00:00.420579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.420544    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c534218e-d003-4a45-82cf-3aafc113fa32" containerName="payload-processing"
Oct 30 22:00:00.420845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.420737    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="179fc693-e7d7-4016-afe4-f45123b21fa9" containerName="payload-pre-processing"
Oct 30 22:00:00.420845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.420754    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c534218e-d003-4a45-82cf-3aafc113fa32" containerName="payload-processing"
Oct 30 22:00:00.425821 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.425793    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"]
Oct 30 22:00:00.427203 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.427177    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:00.434748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.433870    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:00.458811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.447507    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"]
Oct 30 22:00:00.458811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.447583    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"]
Oct 30 22:00:00.471031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.470902    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\""
Oct 30 22:00:00.591120 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.591081    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh\" (UID: \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:00.591388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.591364    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f6s\" (UniqueName: \"kubernetes.io/projected/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-kube-api-access-b6f6s\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh\" (UID: \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:00.591533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.591514    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2\" (UID: \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:00.591606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.591552    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq79m\" (UniqueName: \"kubernetes.io/projected/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-kube-api-access-dq79m\") pod \"maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2\" (UID: \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:00.693310 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.693221    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f6s\" (UniqueName: \"kubernetes.io/projected/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-kube-api-access-b6f6s\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh\" (UID: \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:00.693516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.693349    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2\" (UID: \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:00.693516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.693386    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq79m\" (UniqueName: \"kubernetes.io/projected/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-kube-api-access-dq79m\") pod \"maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2\" (UID: \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:00.693623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.693570    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh\" (UID: \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:00.694363 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.694323    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2\" (UID: \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:00.694557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.694534    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh\" (UID: \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:00.724513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.721141    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f6s\" (UniqueName: \"kubernetes.io/projected/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-kube-api-access-b6f6s\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh\" (UID: \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:00.724513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.721141    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq79m\" (UniqueName: \"kubernetes.io/projected/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-kube-api-access-dq79m\") pod \"maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2\" (UID: \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:00.775494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.774758    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:00.800520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:00.800446    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:01.033628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:01.033581    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:00:01.034099 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:01.033644    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 22:00:01.106818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:01.104191    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"]
Oct 30 22:00:01.192197 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:01.191156    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323ba2ff_6ef1_4923_b4bd_c8c4bc9f680b.slice/crio-46eea2f5806dfc94db022f1b6873e8df510330d5c288190c130fdb7a5444faa8 WatchSource:0}: Error finding container 46eea2f5806dfc94db022f1b6873e8df510330d5c288190c130fdb7a5444faa8: Status 404 returned error can't find the container with id 46eea2f5806dfc94db022f1b6873e8df510330d5c288190c130fdb7a5444faa8
Oct 30 22:00:01.197233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:01.197172    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"]
Oct 30 22:00:01.666340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:01.666257    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179fc693-e7d7-4016-afe4-f45123b21fa9" path="/var/lib/kubelet/pods/179fc693-e7d7-4016-afe4-f45123b21fa9/volumes"
Oct 30 22:00:01.682899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:01.682866    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c534218e-d003-4a45-82cf-3aafc113fa32" path="/var/lib/kubelet/pods/c534218e-d003-4a45-82cf-3aafc113fa32/volumes"
Oct 30 22:00:02.036487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:02.036426    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:00:02.036770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:02.036731    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 22:00:02.037968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:02.037937    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" event={"ID":"1b637f3e-c4ba-4fbe-9e94-7121d1df417b","Type":"ContainerStarted","Data":"16ac31357627e2315db9228ea54b33579f8319b99c578140a012ec3aa69cec81"}
Oct 30 22:00:02.039434 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:02.039412    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" event={"ID":"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b","Type":"ContainerStarted","Data":"46eea2f5806dfc94db022f1b6873e8df510330d5c288190c130fdb7a5444faa8"}
Oct 30 22:00:02.376698 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:02.376581    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:00:02.376698 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:02.376651    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:03.056555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:03.046613    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 22:00:03.056555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:03.046723    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 22:00:03.782437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:03.779398    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 22:00:04.123809 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.121433    4282 generic.go:356] "Generic (PLEG): container finished" podID="dc999841-a85f-49d0-942e-7a09f76df056" containerID="d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528" exitCode=0
Oct 30 22:00:04.123809 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.121574    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" event={"ID":"dc999841-a85f-49d0-942e-7a09f76df056","Type":"ContainerDied","Data":"d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528"}
Oct 30 22:00:04.123809 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.122064    4282 scope.go:122] "RemoveContainer" containerID="d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528"
Oct 30 22:00:04.125974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.124000    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:04.125974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.124092    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:04.125974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.124143    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:04.201135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.200839    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 22:00:04.207674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.206541    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 22:00:04.225284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.222560    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"]
Oct 30 22:00:04.232226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.229099    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"]
Oct 30 22:00:04.232226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.231288    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:04.234801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.234782    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:04.258385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.257809    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e5289f-dockercfg-sd8rj\""
Oct 30 22:00:04.258385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.257822    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e5289f\""
Oct 30 22:00:04.283763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.280396    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"]
Oct 30 22:00:04.291868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.291829    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"]
Oct 30 22:00:04.326026 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.324446    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnf6m\" (UniqueName: \"kubernetes.io/projected/b880fc54-b052-4ba0-a5ea-93ad7aea2419-kube-api-access-dnf6m\") pod \"payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c\" (UID: \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:04.326026 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.324555    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwfxs\" (UniqueName: \"kubernetes.io/projected/aea4f226-8ad6-4b30-af4d-094d681c1a6d-kube-api-access-qwfxs\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5\" (UID: \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:04.326026 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.324692    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b880fc54-b052-4ba0-a5ea-93ad7aea2419-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c\" (UID: \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:04.326026 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.324952    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea4f226-8ad6-4b30-af4d-094d681c1a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5\" (UID: \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:04.585726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.581067    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b880fc54-b052-4ba0-a5ea-93ad7aea2419-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c\" (UID: \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:04.585726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.581306    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea4f226-8ad6-4b30-af4d-094d681c1a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5\" (UID: \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:04.585726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.581440    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnf6m\" (UniqueName: \"kubernetes.io/projected/b880fc54-b052-4ba0-a5ea-93ad7aea2419-kube-api-access-dnf6m\") pod \"payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c\" (UID: \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:04.585726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.581511    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwfxs\" (UniqueName: \"kubernetes.io/projected/aea4f226-8ad6-4b30-af4d-094d681c1a6d-kube-api-access-qwfxs\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5\" (UID: \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:04.605778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.589307    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea4f226-8ad6-4b30-af4d-094d681c1a6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5\" (UID: \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:04.605778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.590128    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b880fc54-b052-4ba0-a5ea-93ad7aea2419-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c\" (UID: \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:04.738740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.738704    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnf6m\" (UniqueName: \"kubernetes.io/projected/b880fc54-b052-4ba0-a5ea-93ad7aea2419-kube-api-access-dnf6m\") pod \"payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c\" (UID: \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:04.796873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.796836    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwfxs\" (UniqueName: \"kubernetes.io/projected/aea4f226-8ad6-4b30-af4d-094d681c1a6d-kube-api-access-qwfxs\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5\" (UID: \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:04.828796 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:04.827869    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:00:04.870047 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.869763    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:04.894786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:04.894756    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:05.135433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:05.135362    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" event={"ID":"dc999841-a85f-49d0-942e-7a09f76df056","Type":"ContainerStarted","Data":"7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d"}
Oct 30 22:00:05.208516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:05.208437    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"]
Oct 30 22:00:05.234568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:05.234543    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"]
Oct 30 22:00:05.236458 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:05.236427    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea4f226_8ad6_4b30_af4d_094d681c1a6d.slice/crio-aa1b15110df64172c1354f4e4571e0ec6a748c9ed29b88e702a6ed346ffb963d WatchSource:0}: Error finding container aa1b15110df64172c1354f4e4571e0ec6a748c9ed29b88e702a6ed346ffb963d: Status 404 returned error can't find the container with id aa1b15110df64172c1354f4e4571e0ec6a748c9ed29b88e702a6ed346ffb963d
Oct 30 22:00:05.287545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:05.287519    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 22:00:05.288386 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:05.288361    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:00:05.288503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:05.288415    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 22:00:06.148083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.148045    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5" event={"ID":"aea4f226-8ad6-4b30-af4d-094d681c1a6d","Type":"ContainerStarted","Data":"3aa09a182ad0c33d3abfbe5be94b555cf71c7f5fe0eeb6cd125a9cffe09786f5"}
Oct 30 22:00:06.148610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.148116    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5" event={"ID":"aea4f226-8ad6-4b30-af4d-094d681c1a6d","Type":"ContainerStarted","Data":"aa1b15110df64172c1354f4e4571e0ec6a748c9ed29b88e702a6ed346ffb963d"}
Oct 30 22:00:06.149243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.149220    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:06.175551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.152735    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c" event={"ID":"b880fc54-b052-4ba0-a5ea-93ad7aea2419","Type":"ContainerStarted","Data":"7b8719a38570f6c8d60c44f0e9f6b3b23f42430fb2ce2d47188d562ec3010b00"}
Oct 30 22:00:06.175551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.152787    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:06.175551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.152810    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c" event={"ID":"b880fc54-b052-4ba0-a5ea-93ad7aea2419","Type":"ContainerStarted","Data":"3d4e2af9501bdd329fc34fe4db03147ef3e09c0d17eb50c39643e64a97db1296"}
Oct 30 22:00:06.196203 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.196031    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5" podStartSLOduration=2.196006845 podStartE2EDuration="2.196006845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:06.189998196 +0000 UTC m=+1224.848094422" watchObservedRunningTime="2026-10-30 22:00:06.196006845 +0000 UTC m=+1224.854103070"
Oct 30 22:00:06.214193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.214136    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c" podStartSLOduration=2.21411712 podStartE2EDuration="2.21411712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:06.210712622 +0000 UTC m=+1224.868808846" watchObservedRunningTime="2026-10-30 22:00:06.21411712 +0000 UTC m=+1224.872213348"
Oct 30 22:00:06.287915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.287884    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:00:06.288097 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:06.287947    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 22:00:07.187778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:07.187737    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" event={"ID":"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b","Type":"ContainerStarted","Data":"9466f36f34f24aaee2fcd744181936c8e0150444b434c6d43969748a5ff05937"}
Oct 30 22:00:07.234500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:07.230354    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" podStartSLOduration=1.882335693 podStartE2EDuration="7.230331293s" totalImagesPullingTime="5.3479956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:07.227956438 +0000 UTC m=+1225.886052662" watchObservedRunningTime="2026-10-30 22:00:07.230331293 +0000 UTC m=+1225.888427516"
Oct 30 22:00:07.310495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:07.309666    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:00:07.310495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:07.309747    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 22:00:07.906924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:07.906889    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"]
Oct 30 22:00:07.907124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:07.907089    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf" podUID="f49d6420-17c2-4528-8a22-ce65fd9dea28" containerName="maas-api" containerID="cri-o://cce12f3ac92595a818b00e42c8ef96531c9d834766f13315b9da6dc636a96235" gracePeriod=30
Oct 30 22:00:08.437490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.305054    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:00:08.437490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.305126    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 22:00:08.437490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.386017    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" event={"ID":"1b637f3e-c4ba-4fbe-9e94-7121d1df417b","Type":"ContainerStarted","Data":"c0725d487c74a8711f570c11793af1f231f24c84a38e2580b9d88cdd52fa0dfd"}
Oct 30 22:00:08.437490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.389234    4282 generic.go:356] "Generic (PLEG): container finished" podID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerID="9466f36f34f24aaee2fcd744181936c8e0150444b434c6d43969748a5ff05937" exitCode=60
Oct 30 22:00:08.437490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.389434    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" event={"ID":"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b","Type":"ContainerDied","Data":"9466f36f34f24aaee2fcd744181936c8e0150444b434c6d43969748a5ff05937"}
Oct 30 22:00:08.437490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.404578    4282 scope.go:122] "RemoveContainer" containerID="9466f36f34f24aaee2fcd744181936c8e0150444b434c6d43969748a5ff05937"
Oct 30 22:00:08.827247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.787588    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" podStartSLOduration=2.974029427 podStartE2EDuration="8.787568587s" totalImagesPullingTime="5.81353916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:08.480045033 +0000 UTC m=+1227.138141257" watchObservedRunningTime="2026-10-30 22:00:08.787568587 +0000 UTC m=+1227.445664811"
Oct 30 22:00:08.848677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.848653    4282 generic.go:356] "Generic (PLEG): container finished" podID="f49d6420-17c2-4528-8a22-ce65fd9dea28" containerID="cce12f3ac92595a818b00e42c8ef96531c9d834766f13315b9da6dc636a96235" exitCode=0
Oct 30 22:00:08.848872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.848725    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf" event={"ID":"f49d6420-17c2-4528-8a22-ce65fd9dea28","Type":"ContainerDied","Data":"cce12f3ac92595a818b00e42c8ef96531c9d834766f13315b9da6dc636a96235"}
Oct 30 22:00:08.933977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:08.933947    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 22:00:09.034536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.034505    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f49d6420-17c2-4528-8a22-ce65fd9dea28-kube-api-access-jstz9\" (UniqueName: \"kubernetes.io/projected/f49d6420-17c2-4528-8a22-ce65fd9dea28-kube-api-access-jstz9\") pod \"f49d6420-17c2-4528-8a22-ce65fd9dea28\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") "
Oct 30 22:00:09.034722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.034545    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-metrics-certs\") pod \"f49d6420-17c2-4528-8a22-ce65fd9dea28\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") "
Oct 30 22:00:09.034722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.034717    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f49d6420-17c2-4528-8a22-ce65fd9dea28-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49d6420-17c2-4528-8a22-ce65fd9dea28-openshift-service-ca\") pod \"f49d6420-17c2-4528-8a22-ce65fd9dea28\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") "
Oct 30 22:00:09.034854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.034819    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-maas-api-tls\") pod \"f49d6420-17c2-4528-8a22-ce65fd9dea28\" (UID: \"f49d6420-17c2-4528-8a22-ce65fd9dea28\") "
Oct 30 22:00:09.035078 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.035047    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49d6420-17c2-4528-8a22-ce65fd9dea28-openshift-service-ca" pod "f49d6420-17c2-4528-8a22-ce65fd9dea28" (UID: "f49d6420-17c2-4528-8a22-ce65fd9dea28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:09.035409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.035389    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49d6420-17c2-4528-8a22-ce65fd9dea28-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:09.062490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.059169    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-metrics-certs" pod "f49d6420-17c2-4528-8a22-ce65fd9dea28" (UID: "f49d6420-17c2-4528-8a22-ce65fd9dea28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:09.095179 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.094761    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-maas-api-tls" pod "f49d6420-17c2-4528-8a22-ce65fd9dea28" (UID: "f49d6420-17c2-4528-8a22-ce65fd9dea28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:09.098781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.098124    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49d6420-17c2-4528-8a22-ce65fd9dea28-kube-api-access-jstz9" pod "f49d6420-17c2-4528-8a22-ce65fd9dea28" (UID: "f49d6420-17c2-4528-8a22-ce65fd9dea28"). InnerVolumeSpecName "kube-api-access-jstz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:09.136591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.136564    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:09.136591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.136590    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jstz9\" (UniqueName: \"kubernetes.io/projected/f49d6420-17c2-4528-8a22-ce65fd9dea28-kube-api-access-jstz9\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:09.136776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.136600    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f49d6420-17c2-4528-8a22-ce65fd9dea28-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:09.288232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.288199    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:00:09.288342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.288260    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 22:00:09.386892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.386768    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"]
Oct 30 22:00:09.387358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:09.387317    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5" podUID="9dff73b1-07de-443b-b2e4-077159317931" containerName="maas-api" containerID="cri-o://38c424312576c23019caad75d8ba1b98313eadc8054dfad1eefc4c1b49576f2d" gracePeriod=30
Oct 30 22:00:10.272548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.266857    4282 generic.go:356] "Generic (PLEG): container finished" podID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerID="c0725d487c74a8711f570c11793af1f231f24c84a38e2580b9d88cdd52fa0dfd" exitCode=60
Oct 30 22:00:10.272548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.267053    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" event={"ID":"1b637f3e-c4ba-4fbe-9e94-7121d1df417b","Type":"ContainerDied","Data":"c0725d487c74a8711f570c11793af1f231f24c84a38e2580b9d88cdd52fa0dfd"}
Oct 30 22:00:10.287987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.280110    4282 scope.go:122] "RemoveContainer" containerID="c0725d487c74a8711f570c11793af1f231f24c84a38e2580b9d88cdd52fa0dfd"
Oct 30 22:00:10.287987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.284458    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:00:10.292777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.292749    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:00:10.292976 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.292953    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 22:00:10.325891 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.323946    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" event={"ID":"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b","Type":"ContainerStarted","Data":"0e400569bc999a791bcf8f97f59e659880cec349571ebd53a72ad47339526b74"}
Oct 30 22:00:10.325891 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.324514    4282 scope.go:122] "RemoveContainer" containerID="0e400569bc999a791bcf8f97f59e659880cec349571ebd53a72ad47339526b74"
Oct 30 22:00:10.325891 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:10.324979    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2_odh-ai-gateway-infra(323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b"
Oct 30 22:00:10.328516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.328144    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf" event={"ID":"f49d6420-17c2-4528-8a22-ce65fd9dea28","Type":"ContainerDied","Data":"874740b4b7b457e8d8c9933efa6800011a92b246c1e5db21f8b93328e9ee37e9"}
Oct 30 22:00:10.328516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.328204    4282 scope.go:122] "RemoveContainer" containerID="cce12f3ac92595a818b00e42c8ef96531c9d834766f13315b9da6dc636a96235"
Oct 30 22:00:10.328516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.328360    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"
Oct 30 22:00:10.339983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.339957    4282 generic.go:356] "Generic (PLEG): container finished" podID="9dff73b1-07de-443b-b2e4-077159317931" containerID="38c424312576c23019caad75d8ba1b98313eadc8054dfad1eefc4c1b49576f2d" exitCode=0
Oct 30 22:00:10.340181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.340159    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5" event={"ID":"9dff73b1-07de-443b-b2e4-077159317931","Type":"ContainerDied","Data":"38c424312576c23019caad75d8ba1b98313eadc8054dfad1eefc4c1b49576f2d"}
Oct 30 22:00:10.623312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.622697    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 22:00:10.709140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.705759    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-metrics-certs\") pod \"9dff73b1-07de-443b-b2e4-077159317931\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") "
Oct 30 22:00:10.709140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.705918    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dff73b1-07de-443b-b2e4-077159317931-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dff73b1-07de-443b-b2e4-077159317931-openshift-service-ca\") pod \"9dff73b1-07de-443b-b2e4-077159317931\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") "
Oct 30 22:00:10.709140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.705984    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-maas-api-tls\") pod \"9dff73b1-07de-443b-b2e4-077159317931\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") "
Oct 30 22:00:10.709140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.706066    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dff73b1-07de-443b-b2e4-077159317931-kube-api-access-ztmfd\" (UniqueName: \"kubernetes.io/projected/9dff73b1-07de-443b-b2e4-077159317931-kube-api-access-ztmfd\") pod \"9dff73b1-07de-443b-b2e4-077159317931\" (UID: \"9dff73b1-07de-443b-b2e4-077159317931\") "
Oct 30 22:00:10.709510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.709150    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dff73b1-07de-443b-b2e4-077159317931-openshift-service-ca" pod "9dff73b1-07de-443b-b2e4-077159317931" (UID: "9dff73b1-07de-443b-b2e4-077159317931"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:10.715528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.713123    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dff73b1-07de-443b-b2e4-077159317931-kube-api-access-ztmfd" pod "9dff73b1-07de-443b-b2e4-077159317931" (UID: "9dff73b1-07de-443b-b2e4-077159317931"). InnerVolumeSpecName "kube-api-access-ztmfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:10.715528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.714859    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-maas-api-tls" pod "9dff73b1-07de-443b-b2e4-077159317931" (UID: "9dff73b1-07de-443b-b2e4-077159317931"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:10.722214 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.722183    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-metrics-certs" pod "9dff73b1-07de-443b-b2e4-077159317931" (UID: "9dff73b1-07de-443b-b2e4-077159317931"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:10.723354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.723334    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"]
Oct 30 22:00:10.731174 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.731154    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf"]
Oct 30 22:00:10.808242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.808210    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dff73b1-07de-443b-b2e4-077159317931-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:10.808242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.808239    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:10.808242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.808249    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztmfd\" (UniqueName: \"kubernetes.io/projected/9dff73b1-07de-443b-b2e4-077159317931-kube-api-access-ztmfd\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:10.808487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:10.808258    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dff73b1-07de-443b-b2e4-077159317931-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:11.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.293290    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:00:11.294027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.293993    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 22:00:11.299583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.299544    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f47f79-757748cf4d-frgdf" podUID="f49d6420-17c2-4528-8a22-ce65fd9dea28" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.102:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:11.615738 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.605971    4282 generic.go:356] "Generic (PLEG): container finished" podID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerID="0e400569bc999a791bcf8f97f59e659880cec349571ebd53a72ad47339526b74" exitCode=60
Oct 30 22:00:11.769605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.760838    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f49d6420-17c2-4528-8a22-ce65fd9dea28" path="/var/lib/kubelet/pods/f49d6420-17c2-4528-8a22-ce65fd9dea28/volumes"
Oct 30 22:00:11.769605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.761976    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" event={"ID":"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b","Type":"ContainerDied","Data":"0e400569bc999a791bcf8f97f59e659880cec349571ebd53a72ad47339526b74"}
Oct 30 22:00:11.769605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.762062    4282 scope.go:122] "RemoveContainer" containerID="9466f36f34f24aaee2fcd744181936c8e0150444b434c6d43969748a5ff05937"
Oct 30 22:00:11.795488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.778780    4282 scope.go:122] "RemoveContainer" containerID="0e400569bc999a791bcf8f97f59e659880cec349571ebd53a72ad47339526b74"
Oct 30 22:00:11.795488 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:11.779457    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2_odh-ai-gateway-infra(323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b"
Oct 30 22:00:11.843129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.837907    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"
Oct 30 22:00:11.847968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.847857    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5" event={"ID":"9dff73b1-07de-443b-b2e4-077159317931","Type":"ContainerDied","Data":"2f92223dec630997ceddfbcc46c0ee4d467150f32e10371e07a2cfa4903228f6"}
Oct 30 22:00:11.904501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.893363    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" event={"ID":"1b637f3e-c4ba-4fbe-9e94-7121d1df417b","Type":"ContainerStarted","Data":"6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568"}
Oct 30 22:00:11.904501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:11.893886    4282 scope.go:122] "RemoveContainer" containerID="6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568"
Oct 30 22:00:11.904501 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:11.894336    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh_odh-ai-gateway-infra(1b637f3e-c4ba-4fbe-9e94-7121d1df417b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b"
Oct 30 22:00:12.034061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:12.034043    4282 scope.go:122] "RemoveContainer" containerID="38c424312576c23019caad75d8ba1b98313eadc8054dfad1eefc4c1b49576f2d"
Oct 30 22:00:12.045154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:12.045096    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"]
Oct 30 22:00:12.052052 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:12.052023    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8d2c42-644bf67649-57mt5"]
Oct 30 22:00:12.115112 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:12.115075    4282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dff73b1_07de_443b_b2e4_077159317931.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dff73b1_07de_443b_b2e4_077159317931.slice/crio-2f92223dec630997ceddfbcc46c0ee4d467150f32e10371e07a2cfa4903228f6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b637f3e_c4ba_4fbe_9e94_7121d1df417b.slice/crio-conmon-6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:00:12.405195 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:12.402614    4282 patch_prober.go:29] interesting pod/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body=
Oct 30 22:00:12.405195 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:12.402675    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused"
Oct 30 22:00:13.004501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.003503    4282 scope.go:122] "RemoveContainer" containerID="0e400569bc999a791bcf8f97f59e659880cec349571ebd53a72ad47339526b74"
Oct 30 22:00:13.004501 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:13.003967    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2_odh-ai-gateway-infra(323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b"
Oct 30 22:00:13.013532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.009792    4282 generic.go:356] "Generic (PLEG): container finished" podID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerID="6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568" exitCode=60
Oct 30 22:00:13.013532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.009848    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" event={"ID":"1b637f3e-c4ba-4fbe-9e94-7121d1df417b","Type":"ContainerDied","Data":"6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568"}
Oct 30 22:00:13.013532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.009901    4282 scope.go:122] "RemoveContainer" containerID="c0725d487c74a8711f570c11793af1f231f24c84a38e2580b9d88cdd52fa0dfd"
Oct 30 22:00:13.013532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.013433    4282 scope.go:122] "RemoveContainer" containerID="6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568"
Oct 30 22:00:13.014267 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:13.013896    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh_odh-ai-gateway-infra(1b637f3e-c4ba-4fbe-9e94-7121d1df417b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b"
Oct 30 22:00:13.293317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.293235    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 22:00:13.293711 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.293685    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 22:00:13.294542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.294515    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 22:00:13.502624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:13.502580    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dff73b1-07de-443b-b2e4-077159317931" path="/var/lib/kubelet/pods/9dff73b1-07de-443b-b2e4-077159317931/volumes"
Oct 30 22:00:14.107490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:14.105452    4282 scope.go:122] "RemoveContainer" containerID="6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568"
Oct 30 22:00:14.107490 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:14.105956    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh_odh-ai-gateway-infra(1b637f3e-c4ba-4fbe-9e94-7121d1df417b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b"
Oct 30 22:00:15.019993 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.019958    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"]
Oct 30 22:00:15.020701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.020682    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f49d6420-17c2-4528-8a22-ce65fd9dea28" containerName="maas-api"
Oct 30 22:00:15.020701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.020698    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f49d6420-17c2-4528-8a22-ce65fd9dea28" containerName="maas-api"
Oct 30 22:00:15.020867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.020729    4282 cpu_manager.go:414] "containerMap: removing container" podUID="9dff73b1-07de-443b-b2e4-077159317931" containerName="maas-api"
Oct 30 22:00:15.020867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.020734    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dff73b1-07de-443b-b2e4-077159317931" containerName="maas-api"
Oct 30 22:00:15.020867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.020821    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f49d6420-17c2-4528-8a22-ce65fd9dea28" containerName="maas-api"
Oct 30 22:00:15.020867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.020834    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dff73b1-07de-443b-b2e4-077159317931" containerName="maas-api"
Oct 30 22:00:15.026327 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.026192    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.037098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.037076    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e5289f-serving-cert\""
Oct 30 22:00:15.038938 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.038908    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.039079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.039064    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwk9\" (UniqueName: \"kubernetes.io/projected/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-kube-api-access-6kwk9\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.039210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.039193    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.039283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.039227    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-metrics-certs\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.045206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.045185    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"]
Oct 30 22:00:15.140933 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.140822    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.140933 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.140893    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-metrics-certs\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.141205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.141032    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.141689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.141665    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwk9\" (UniqueName: \"kubernetes.io/projected/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-kube-api-access-6kwk9\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.142046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.142021    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.144024 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.144001    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-metrics-certs\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.144396 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.144377    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.181967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.181930    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwk9\" (UniqueName: \"kubernetes.io/projected/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-kube-api-access-6kwk9\") pod \"maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.363843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.358836    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:15.940543 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:15.937990    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"]
Oct 30 22:00:16.121188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:16.121109    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd" event={"ID":"c780ada2-2ab1-4eb3-bd75-939a40a1eef7","Type":"ContainerStarted","Data":"b4d81d58539e0e9d0b2d3182b5ef43078954e74ef0847d5096a67c98042595c7"}
Oct 30 22:00:16.734898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:16.734817    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"]
Oct 30 22:00:16.735346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:16.735314    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy" containerID="cri-o://cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce" gracePeriod=30
Oct 30 22:00:17.171093 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:17.171049    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd" event={"ID":"c780ada2-2ab1-4eb3-bd75-939a40a1eef7","Type":"ContainerStarted","Data":"772e55efbac1218802d40f412ab110780695841512664fdea7083c840e4dd7b0"}
Oct 30 22:00:17.189494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:17.172867    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:17.215495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:17.205914    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:17.215495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:17.212890    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd" podStartSLOduration=2.662725052 podStartE2EDuration="3.212871998s" totalImagesPullingTime="550.146946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:17.209780741 +0000 UTC m=+1235.867876979" watchObservedRunningTime="2026-10-30 22:00:17.212871998 +0000 UTC m=+1235.870968222"
Oct 30 22:00:18.855048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:18.855019    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:19.984207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:19.984161    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"]
Oct 30 22:00:19.992775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:19.992738    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:19.998953 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:19.998932    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ecffad-openshift-default-dockercfg-k4fpn\""
Oct 30 22:00:20.000813 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.000788    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"]
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.156569    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rbpp\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-kube-api-access-6rbpp\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.156725    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-token\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.156769    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-data\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.156854    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-envoy\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.156913    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-credential-socket\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.156972    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-socket\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.157005    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-certs\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.157163    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-podinfo\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.166492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.157225    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istiod-ca-cert\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.263721    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-token\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.263791    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-data\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.263879    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-envoy\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.263928    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-credential-socket\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.263973    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-socket\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.264010    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-certs\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.264169    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-podinfo\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.264220    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istiod-ca-cert\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.264547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.264277    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rbpp\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-kube-api-access-6rbpp\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.285493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.265369    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-data\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.301490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.299860    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-credential-socket\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.301490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.300118    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-socket\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.301490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.300370    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-certs\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.341496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.327989    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istiod-ca-cert\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.371530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.370813    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-token\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.396502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.396415    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-podinfo\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.426391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.426355    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-envoy\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.435525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.428161    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rbpp\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-kube-api-access-6rbpp\") pod \"e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") " pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.612218 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.612125    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:20.917400 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:20.917349    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519c0cc2_aeef_4a66_b158_f6e703e5d0da.slice/crio-46a030ba06c535341654056986462534cfcf776c97d997b10130cd31e9c75250 WatchSource:0}: Error finding container 46a030ba06c535341654056986462534cfcf776c97d997b10130cd31e9c75250: Status 404 returned error can't find the container with id 46a030ba06c535341654056986462534cfcf776c97d997b10130cd31e9c75250
Oct 30 22:00:20.922078 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.922028    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:20.922299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.922131    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:20.922299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.922177    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:20.931828 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:20.931798    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"]
Oct 30 22:00:21.219730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:21.219686    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:00:21.220197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:21.219762    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:21.345228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:21.345196    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" event={"ID":"519c0cc2-aeef-4a66-b158-f6e703e5d0da","Type":"ContainerStarted","Data":"9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002"}
Oct 30 22:00:21.345408 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:21.345254    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" event={"ID":"519c0cc2-aeef-4a66-b158-f6e703e5d0da","Type":"ContainerStarted","Data":"46a030ba06c535341654056986462534cfcf776c97d997b10130cd31e9c75250"}
Oct 30 22:00:21.371023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:21.370965    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" podStartSLOduration=2.370948876 podStartE2EDuration="2.370948876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:21.367741284 +0000 UTC m=+1240.025837509" watchObservedRunningTime="2026-10-30 22:00:21.370948876 +0000 UTC m=+1240.029045138"
Oct 30 22:00:21.460856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:21.460773    4282 scope.go:122] "RemoveContainer" containerID="0e400569bc999a791bcf8f97f59e659880cec349571ebd53a72ad47339526b74"
Oct 30 22:00:21.622495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:21.617034    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:22.078207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.078185    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 22:00:22.164288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.164245    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-credential-socket\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.185492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185213    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-kube-api-access-m78xn\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-kube-api-access-m78xn\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.185492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185384    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-certs\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.185492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185418    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-socket\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.185492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185498    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-envoy\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.185854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185550    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-token\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-token\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.185854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185585    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-data\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.185854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185694    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68171810-3eb8-4c3a-a758-f11eaafe5082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68171810-3eb8-4c3a-a758-f11eaafe5082-istiod-ca-cert\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.185854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185785    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-podinfo\") pod \"68171810-3eb8-4c3a-a758-f11eaafe5082\" (UID: \"68171810-3eb8-4c3a-a758-f11eaafe5082\") "
Oct 30 22:00:22.186731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.185080    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-credential-socket" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:22.186731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.186368    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-socket" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:22.187716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.187008    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.187716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.187037    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.192818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.192783    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-certs" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:22.206544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.193085    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-data" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:22.206544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.193371    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68171810-3eb8-4c3a-a758-f11eaafe5082-istiod-ca-cert" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:22.206544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.200339    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-token" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:22.232496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.227844    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-podinfo" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:22.248752 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.248695    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-kube-api-access-m78xn" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "kube-api-access-m78xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:22.250716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.250663    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-envoy" pod "68171810-3eb8-4c3a-a758-f11eaafe5082" (UID: "68171810-3eb8-4c3a-a758-f11eaafe5082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:22.288691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.288664    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.288691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.288689    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.288856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.288699    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68171810-3eb8-4c3a-a758-f11eaafe5082-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.288856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.288708    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.288856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.288717    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m78xn\" (UniqueName: \"kubernetes.io/projected/68171810-3eb8-4c3a-a758-f11eaafe5082-kube-api-access-m78xn\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.288856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.288725    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.288856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.288733    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68171810-3eb8-4c3a-a758-f11eaafe5082-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.354769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.354692    4282 generic.go:356] "Generic (PLEG): container finished" podID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerID="cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce" exitCode=0
Oct 30 22:00:22.354769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.354760    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"
Oct 30 22:00:22.354993 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.354792    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" event={"ID":"68171810-3eb8-4c3a-a758-f11eaafe5082","Type":"ContainerDied","Data":"cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce"}
Oct 30 22:00:22.354993 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.354861    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj" event={"ID":"68171810-3eb8-4c3a-a758-f11eaafe5082","Type":"ContainerDied","Data":"0cb9ee38f08ed3ba3961543353b0e79586b80a61619fd12b472b1e0b86262916"}
Oct 30 22:00:22.354993 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.354888    4282 scope.go:122] "RemoveContainer" containerID="cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce"
Oct 30 22:00:22.361237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.361209    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" event={"ID":"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b","Type":"ContainerStarted","Data":"6c5687be80539751c37161e297972bb93bcb1809424ba917edb4f6603bb7de44"}
Oct 30 22:00:22.371720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.371702    4282 scope.go:122] "RemoveContainer" containerID="b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18"
Oct 30 22:00:22.381369 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.381349    4282 scope.go:122] "RemoveContainer" containerID="cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce"
Oct 30 22:00:22.382179 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:22.381760    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce\": container with ID starting with cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce not found: ID does not exist" containerID="cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce"
Oct 30 22:00:22.382179 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.381811    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce"} err="failed to get container status \"cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce\": rpc error: code = NotFound desc = could not find container \"cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce\": container with ID starting with cb75da1db72a8637eaabea7eff57705264957f054a758e0e5893ee99bb7f62ce not found: ID does not exist"
Oct 30 22:00:22.382179 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.381833    4282 scope.go:122] "RemoveContainer" containerID="b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18"
Oct 30 22:00:22.382179 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:22.382140    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18\": container with ID starting with b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18 not found: ID does not exist" containerID="b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18"
Oct 30 22:00:22.382440 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.382165    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18"} err="failed to get container status \"b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18\": rpc error: code = NotFound desc = could not find container \"b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18\": container with ID starting with b55ee2b971f148abb1a1deab574bcf491479da679d1b69c22783c5ec1739ad18 not found: ID does not exist"
Oct 30 22:00:22.400782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.400758    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"]
Oct 30 22:00:22.409827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.409801    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f47f79-openshift-default-7b47fc8649-7zbbj"]
Oct 30 22:00:22.619014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:22.618945    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:23.405806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.405754    4282 generic.go:356] "Generic (PLEG): container finished" podID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerID="6c5687be80539751c37161e297972bb93bcb1809424ba917edb4f6603bb7de44" exitCode=60
Oct 30 22:00:23.406360 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.406314    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" event={"ID":"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b","Type":"ContainerDied","Data":"6c5687be80539751c37161e297972bb93bcb1809424ba917edb4f6603bb7de44"}
Oct 30 22:00:23.406582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.406564    4282 scope.go:122] "RemoveContainer" containerID="0e400569bc999a791bcf8f97f59e659880cec349571ebd53a72ad47339526b74"
Oct 30 22:00:23.407073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.407051    4282 scope.go:122] "RemoveContainer" containerID="6c5687be80539751c37161e297972bb93bcb1809424ba917edb4f6603bb7de44"
Oct 30 22:00:23.407424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.407407    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:23.407631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.407616    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"]
Oct 30 22:00:23.407764 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:23.407421    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2_odh-ai-gateway-infra(323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b"
Oct 30 22:00:23.410056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.409764    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:00:23.456697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.456671    4282 scope.go:122] "RemoveContainer" containerID="6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568"
Oct 30 22:00:23.465796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.465770    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" path="/var/lib/kubelet/pods/68171810-3eb8-4c3a-a758-f11eaafe5082/volumes"
Oct 30 22:00:23.539226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.539192    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"]
Oct 30 22:00:23.539427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.539406    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy" containerID="cri-o://46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a" gracePeriod=30
Oct 30 22:00:23.543274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.543248    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"]
Oct 30 22:00:23.543520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:23.543492    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy" containerID="cri-o://ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550" gracePeriod=30
Oct 30 22:00:24.303493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.301692    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:00:24.418457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.418428    4282 generic.go:356] "Generic (PLEG): container finished" podID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerID="cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e" exitCode=0
Oct 30 22:00:24.418926 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.418526    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" event={"ID":"7646623d-2c6a-4b60-90f8-31b67cb69c05","Type":"ContainerDied","Data":"cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e"}
Oct 30 22:00:24.419056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.419042    4282 scope.go:122] "RemoveContainer" containerID="cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e"
Oct 30 22:00:24.420879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.420855    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" event={"ID":"1b637f3e-c4ba-4fbe-9e94-7121d1df417b","Type":"ContainerStarted","Data":"6c94aae2bcb90c4a4c08c23b3ec73a0c4a068ffb908518a44a88e367ec15ee90"}
Oct 30 22:00:24.421146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.421100    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:24.421278 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.421190    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:24.421278 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.421235    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:24.758671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.758645    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:24.790560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.790533    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-maas-api-service-ca\") pod \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\" (UID: \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\") "
Oct 30 22:00:24.790749 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.790592    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-kube-api-access-dq79m\" (UniqueName: \"kubernetes.io/projected/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-kube-api-access-dq79m\") pod \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\" (UID: \"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b\") "
Oct 30 22:00:24.790966 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.790938    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-maas-api-service-ca" pod "323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" (UID: "323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:24.791461 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.791400    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:24.793004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.792980    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-kube-api-access-dq79m" pod "323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" (UID: "323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b"). InnerVolumeSpecName "kube-api-access-dq79m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:24.913564 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:24.895226    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq79m\" (UniqueName: \"kubernetes.io/projected/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b-kube-api-access-dq79m\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:25.332352 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.332322    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"]
Oct 30 22:00:25.332562 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.332542    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy" containerID="cri-o://66794b45ffe05dbd78307ae0412993aa70de6d6b435426c439ca788ef3ac090a" gracePeriod=30
Oct 30 22:00:25.441511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.441367    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" event={"ID":"7646623d-2c6a-4b60-90f8-31b67cb69c05","Type":"ContainerStarted","Data":"2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6"}
Oct 30 22:00:25.444239 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.444208    4282 generic.go:356] "Generic (PLEG): container finished" podID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerID="6c94aae2bcb90c4a4c08c23b3ec73a0c4a068ffb908518a44a88e367ec15ee90" exitCode=60
Oct 30 22:00:25.444418 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.444374    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" event={"ID":"1b637f3e-c4ba-4fbe-9e94-7121d1df417b","Type":"ContainerDied","Data":"6c94aae2bcb90c4a4c08c23b3ec73a0c4a068ffb908518a44a88e367ec15ee90"}
Oct 30 22:00:25.445625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.445605    4282 scope.go:122] "RemoveContainer" containerID="6634dce071580e91f982bd56febf33fa4fc83bcd430f02662f9b77b64c11a568"
Oct 30 22:00:25.445797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.445706    4282 scope.go:122] "RemoveContainer" containerID="6c94aae2bcb90c4a4c08c23b3ec73a0c4a068ffb908518a44a88e367ec15ee90"
Oct 30 22:00:25.446142 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:25.446112    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh_odh-ai-gateway-infra(1b637f3e-c4ba-4fbe-9e94-7121d1df417b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b"
Oct 30 22:00:25.447130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.446919    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"
Oct 30 22:00:25.447919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.447897    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2" event={"ID":"323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b","Type":"ContainerDied","Data":"46eea2f5806dfc94db022f1b6873e8df510330d5c288190c130fdb7a5444faa8"}
Oct 30 22:00:25.469212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.469187    4282 scope.go:122] "RemoveContainer" containerID="6c5687be80539751c37161e297972bb93bcb1809424ba917edb4f6603bb7de44"
Oct 30 22:00:25.516885 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.516853    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"]
Oct 30 22:00:25.529899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.529869    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"]
Oct 30 22:00:25.534131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.534108    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-0b3fe4-29889960-stpr2"]
Oct 30 22:00:25.607297 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607216    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"]
Oct 30 22:00:25.607905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607891    4282 cpu_manager.go:414] "containerMap: removing container" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:25.607967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607907    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:25.607967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607926    4282 cpu_manager.go:414] "containerMap: removing container" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy"
Oct 30 22:00:25.607967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607931    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy"
Oct 30 22:00:25.607967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607945    4282 cpu_manager.go:414] "containerMap: removing container" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy"
Oct 30 22:00:25.607967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607950    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy"
Oct 30 22:00:25.607967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607962    4282 cpu_manager.go:414] "containerMap: removing container" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:25.607967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.607966    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:25.608173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.608080    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:25.608173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.608088    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:25.608173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.608095    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy"
Oct 30 22:00:25.608173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.608103    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="68171810-3eb8-4c3a-a758-f11eaafe5082" containerName="istio-proxy"
Oct 30 22:00:25.608173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.608109    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:25.614039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.614020    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:25.618756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.618730    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ecffad-dockercfg-8vb22\""
Oct 30 22:00:25.618756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.618749    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ecffad\""
Oct 30 22:00:25.627028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.627001    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"]
Oct 30 22:00:25.756494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.741191    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v48js\" (UniqueName: \"kubernetes.io/projected/200c1579-38d3-4168-8cb1-81de4feb4793-kube-api-access-v48js\") pod \"payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4\" (UID: \"200c1579-38d3-4168-8cb1-81de4feb4793\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:25.756494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.741368    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200c1579-38d3-4168-8cb1-81de4feb4793-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4\" (UID: \"200c1579-38d3-4168-8cb1-81de4feb4793\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:25.940545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.924092    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200c1579-38d3-4168-8cb1-81de4feb4793-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4\" (UID: \"200c1579-38d3-4168-8cb1-81de4feb4793\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:25.970516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.970458    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v48js\" (UniqueName: \"kubernetes.io/projected/200c1579-38d3-4168-8cb1-81de4feb4793-kube-api-access-v48js\") pod \"payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4\" (UID: \"200c1579-38d3-4168-8cb1-81de4feb4793\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:26.083494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:25.942222    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200c1579-38d3-4168-8cb1-81de4feb4793-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4\" (UID: \"200c1579-38d3-4168-8cb1-81de4feb4793\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:26.139561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.098106    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 22:00:26.139561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.115483    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v48js\" (UniqueName: \"kubernetes.io/projected/200c1579-38d3-4168-8cb1-81de4feb4793-kube-api-access-v48js\") pod \"payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4\" (UID: \"200c1579-38d3-4168-8cb1-81de4feb4793\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:26.139561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.126004    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:00:26.139561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.126142    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:00:26.260940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.260599    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:26.687427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.687401    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"]
Oct 30 22:00:26.689567 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:26.689540    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod200c1579_38d3_4168_8cb1_81de4feb4793.slice/crio-6b6c5e7ad012ca7a0d0190eaa62cb1ce0d5adae70407b413c6fe458a0868e72a WatchSource:0}: Error finding container 6b6c5e7ad012ca7a0d0190eaa62cb1ce0d5adae70407b413c6fe458a0868e72a: Status 404 returned error can't find the container with id 6b6c5e7ad012ca7a0d0190eaa62cb1ce0d5adae70407b413c6fe458a0868e72a
Oct 30 22:00:26.817649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.817623    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:26.909356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.909254    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-maas-api-service-ca\") pod \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\" (UID: \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\") "
Oct 30 22:00:26.924676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.924646    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-kube-api-access-b6f6s\" (UniqueName: \"kubernetes.io/projected/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-kube-api-access-b6f6s\") pod \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\" (UID: \"1b637f3e-c4ba-4fbe-9e94-7121d1df417b\") "
Oct 30 22:00:26.932515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.924509    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-maas-api-service-ca" pod "1b637f3e-c4ba-4fbe-9e94-7121d1df417b" (UID: "1b637f3e-c4ba-4fbe-9e94-7121d1df417b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:26.950904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:26.946567    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-kube-api-access-b6f6s" pod "1b637f3e-c4ba-4fbe-9e94-7121d1df417b" (UID: "1b637f3e-c4ba-4fbe-9e94-7121d1df417b"). InnerVolumeSpecName "kube-api-access-b6f6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:27.027665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.027636    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:27.027665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.027663    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6f6s\" (UniqueName: \"kubernetes.io/projected/1b637f3e-c4ba-4fbe-9e94-7121d1df417b-kube-api-access-b6f6s\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:27.080320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.080291    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:00:27.080534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.080340    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:00:27.484718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.484684    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" path="/var/lib/kubelet/pods/323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b/volumes"
Oct 30 22:00:27.503039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.503006    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4" event={"ID":"200c1579-38d3-4168-8cb1-81de4feb4793","Type":"ContainerStarted","Data":"d199360064462638e773d75ec3d97a832a21aa7c38f9103c7a2a643dd85f45ed"}
Oct 30 22:00:27.503196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.503061    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:27.503196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.503086    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4" event={"ID":"200c1579-38d3-4168-8cb1-81de4feb4793","Type":"ContainerStarted","Data":"6b6c5e7ad012ca7a0d0190eaa62cb1ce0d5adae70407b413c6fe458a0868e72a"}
Oct 30 22:00:27.504449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.504424    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh" event={"ID":"1b637f3e-c4ba-4fbe-9e94-7121d1df417b","Type":"ContainerDied","Data":"16ac31357627e2315db9228ea54b33579f8319b99c578140a012ec3aa69cec81"}
Oct 30 22:00:27.504449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.504446    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"
Oct 30 22:00:27.504659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.504523    4282 scope.go:122] "RemoveContainer" containerID="6c94aae2bcb90c4a4c08c23b3ec73a0c4a068ffb908518a44a88e367ec15ee90"
Oct 30 22:00:27.524781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.524732    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4" podStartSLOduration=2.524718996 podStartE2EDuration="2.524718996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:27.51919665 +0000 UTC m=+1246.177292872" watchObservedRunningTime="2026-10-30 22:00:27.524718996 +0000 UTC m=+1246.182815270"
Oct 30 22:00:27.538405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.538375    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"]
Oct 30 22:00:27.542261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:27.542236    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889960-zpzsh"]
Oct 30 22:00:28.078295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:28.078256    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:00:28.078745 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:28.078307    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:00:29.082684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.082019    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:00:29.082684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.082079    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:00:29.092165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.092083    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 22:00:29.119702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.119678    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-socket\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.119817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.119765    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-kube-api-access-n2xwm\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-kube-api-access-n2xwm\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.119887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.119864    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-data\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.119952 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.119937    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-credential-socket\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.120018 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.119951    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-socket" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.120084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120062    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-envoy\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.120161 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120147    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-istio-token\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-istio-token\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.120219 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120140    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-data" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.120219 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120190    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75810534-d64f-4003-89e9-1222cb927848-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75810534-d64f-4003-89e9-1222cb927848-istiod-ca-cert\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.120219 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120204    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-credential-socket" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.120413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120307    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-certs\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.120413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120371    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75810534-d64f-4003-89e9-1222cb927848-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75810534-d64f-4003-89e9-1222cb927848-istio-podinfo\") pod \"75810534-d64f-4003-89e9-1222cb927848\" (UID: \"75810534-d64f-4003-89e9-1222cb927848\") "
Oct 30 22:00:29.120774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120648    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75810534-d64f-4003-89e9-1222cb927848-istiod-ca-cert" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:29.120854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.120752    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-certs" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.121357 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.121331    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.121456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.121361    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.121456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.121377    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75810534-d64f-4003-89e9-1222cb927848-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.121456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.121393    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.121456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.121408    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.122763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.122743    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-istio-token" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:29.122971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.122954    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 22:00:29.123113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.123077    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-envoy" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.123534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.123508    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-kube-api-access-n2xwm" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "kube-api-access-n2xwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:29.125879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.125834    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75810534-d64f-4003-89e9-1222cb927848-istio-podinfo" pod "75810534-d64f-4003-89e9-1222cb927848" (UID: "75810534-d64f-4003-89e9-1222cb927848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:29.221978 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.221947    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a21738-5faf-4cea-899c-3be3bc5c62f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a21738-5faf-4cea-899c-3be3bc5c62f2-istiod-ca-cert\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.221989    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-token\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-token\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222045    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-certs\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222145    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-socket\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222181    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-kube-api-access-gjxkw\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-kube-api-access-gjxkw\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222220    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-credential-socket\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222258    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-envoy\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222295    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-data\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222334    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-podinfo\") pod \"97a21738-5faf-4cea-899c-3be3bc5c62f2\" (UID: \"97a21738-5faf-4cea-899c-3be3bc5c62f2\") "
Oct 30 22:00:29.222423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222346    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a21738-5faf-4cea-899c-3be3bc5c62f2-istiod-ca-cert" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:29.222731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222602    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-certs" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.222731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222617    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-socket" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.222731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.222665    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-data" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.223046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223003    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-credential-socket" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.223350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223231    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75810534-d64f-4003-89e9-1222cb927848-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.223350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223254    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.223350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223269    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.223350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223282    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.223350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223295    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.223350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223308    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75810534-d64f-4003-89e9-1222cb927848-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.223350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223323    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2xwm\" (UniqueName: \"kubernetes.io/projected/75810534-d64f-4003-89e9-1222cb927848-kube-api-access-n2xwm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.223350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223338    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97a21738-5faf-4cea-899c-3be3bc5c62f2-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.223811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.223351    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.224716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.224692    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-token" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:29.224908 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.224885    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-podinfo" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:29.225003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.224906    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-envoy" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:29.225003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.224941    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-kube-api-access-gjxkw" pod "97a21738-5faf-4cea-899c-3be3bc5c62f2" (UID: "97a21738-5faf-4cea-899c-3be3bc5c62f2"). InnerVolumeSpecName "kube-api-access-gjxkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:29.325217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.325189    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.325217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.325214    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjxkw\" (UniqueName: \"kubernetes.io/projected/97a21738-5faf-4cea-899c-3be3bc5c62f2-kube-api-access-gjxkw\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.325217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.325223    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.325420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.325234    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97a21738-5faf-4cea-899c-3be3bc5c62f2-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:29.486061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.486007    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" path="/var/lib/kubelet/pods/1b637f3e-c4ba-4fbe-9e94-7121d1df417b/volumes"
Oct 30 22:00:29.644957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.644924    4282 generic.go:356] "Generic (PLEG): container finished" podID="75810534-d64f-4003-89e9-1222cb927848" containerID="ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550" exitCode=0
Oct 30 22:00:29.645226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.644972    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"
Oct 30 22:00:29.645743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.645002    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" event={"ID":"75810534-d64f-4003-89e9-1222cb927848","Type":"ContainerDied","Data":"ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550"}
Oct 30 22:00:29.645743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.645702    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv" event={"ID":"75810534-d64f-4003-89e9-1222cb927848","Type":"ContainerDied","Data":"81cf9184f5f0a6da53e4d4524014c3237f6b6d6de3375ce05a0e005ff4d5a492"}
Oct 30 22:00:29.645743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.645739    4282 scope.go:122] "RemoveContainer" containerID="ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550"
Oct 30 22:00:29.647086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.647060    4282 generic.go:356] "Generic (PLEG): container finished" podID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerID="46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a" exitCode=0
Oct 30 22:00:29.647810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.647218    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"
Oct 30 22:00:29.647810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.647235    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" event={"ID":"97a21738-5faf-4cea-899c-3be3bc5c62f2","Type":"ContainerDied","Data":"46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a"}
Oct 30 22:00:29.647810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.647293    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z" event={"ID":"97a21738-5faf-4cea-899c-3be3bc5c62f2","Type":"ContainerDied","Data":"47451860c520b3b0ff37c4f764b6644b91f08b7621b93746fa680b00f2c5b50d"}
Oct 30 22:00:29.670059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.670034    4282 scope.go:122] "RemoveContainer" containerID="ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550"
Oct 30 22:00:29.672492 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:29.670520    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550\": container with ID starting with ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550 not found: ID does not exist" containerID="ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550"
Oct 30 22:00:29.672709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.672649    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550"} err="failed to get container status \"ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550\": rpc error: code = NotFound desc = could not find container \"ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550\": container with ID starting with ec971b337491fdaded60bf8900bb626b9f91365e592fcc751073a3fd6a9cf550 not found: ID does not exist"
Oct 30 22:00:29.672709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.672683    4282 scope.go:122] "RemoveContainer" containerID="46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a"
Oct 30 22:00:29.696938 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.696548    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"]
Oct 30 22:00:29.696938 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.696838    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-rsp4z"]
Oct 30 22:00:29.697557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.697535    4282 scope.go:122] "RemoveContainer" containerID="46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a"
Oct 30 22:00:29.697844 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:29.697825    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a\": container with ID starting with 46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a not found: ID does not exist" containerID="46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a"
Oct 30 22:00:29.697901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.697853    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a"} err="failed to get container status \"46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a\": rpc error: code = NotFound desc = could not find container \"46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a\": container with ID starting with 46983285d25c04f5e6708fc64982cc9c7a0c130b7fe3d59cfcee9c4b2831c42a not found: ID does not exist"
Oct 30 22:00:29.711567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.711546    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"]
Oct 30 22:00:29.717964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.717942    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-gw7zv"]
Oct 30 22:00:29.771653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:29.768594    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"]
Oct 30 22:00:30.120546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:30.108581    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:00:30.120546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:30.108644    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:00:30.987671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:30.987486    4282 generic.go:356] "Generic (PLEG): container finished" podID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerID="66794b45ffe05dbd78307ae0412993aa70de6d6b435426c439ca788ef3ac090a" exitCode=0
Oct 30 22:00:30.987671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:30.987608    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" event={"ID":"4918fb2b-e3d4-4584-93f0-46cbdad0e656","Type":"ContainerDied","Data":"66794b45ffe05dbd78307ae0412993aa70de6d6b435426c439ca788ef3ac090a"}
Oct 30 22:00:31.080958 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.080920    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:00:31.081220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.081182    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 30 22:00:31.129104 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.129080    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 22:00:31.218826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.218792    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-envoy\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.219076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.218891    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-token\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-token\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.219076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.218921    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-credential-socket\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.219262 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.219234    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-credential-socket" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:31.219444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.219295    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-certs\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.219444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.219335    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-data\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.219760 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.219694    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-podinfo\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.220038 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.220013    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-kube-api-access-4pkzf\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-kube-api-access-4pkzf\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.220186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.220168    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istiod-ca-cert\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.220253 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.220234    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-socket\") pod \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\" (UID: \"4918fb2b-e3d4-4584-93f0-46cbdad0e656\") "
Oct 30 22:00:31.220628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.220601    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-certs" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:31.220995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.220971    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-socket" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:31.221205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.221184    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.221319 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.221305    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.221400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.221220    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-data" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:31.221982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.221959    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istiod-ca-cert" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:31.222085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.222026    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-envoy" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:31.223934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.223909    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-token" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:31.225016 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.224321    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-kube-api-access-4pkzf" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "kube-api-access-4pkzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:31.225016 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.224452    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-podinfo" pod "4918fb2b-e3d4-4584-93f0-46cbdad0e656" (UID: "4918fb2b-e3d4-4584-93f0-46cbdad0e656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:31.323359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.323262    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.323359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.323296    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.323359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.323310    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.323359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.323321    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.323359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.323333    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.323359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.323346    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4918fb2b-e3d4-4584-93f0-46cbdad0e656-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.323359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.323359    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pkzf\" (UniqueName: \"kubernetes.io/projected/4918fb2b-e3d4-4584-93f0-46cbdad0e656-kube-api-access-4pkzf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:31.480497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.471294    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75810534-d64f-4003-89e9-1222cb927848" path="/var/lib/kubelet/pods/75810534-d64f-4003-89e9-1222cb927848/volumes"
Oct 30 22:00:31.481622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:31.481579    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" path="/var/lib/kubelet/pods/97a21738-5faf-4cea-899c-3be3bc5c62f2/volumes"
Oct 30 22:00:32.005330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.005298    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"
Oct 30 22:00:32.005583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.005294    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr" event={"ID":"4918fb2b-e3d4-4584-93f0-46cbdad0e656","Type":"ContainerDied","Data":"e976cbf5eeb847eed99838d01afbbf544bd01fd0bb39e3fe748a5c2e8df5e0fd"}
Oct 30 22:00:32.005583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.005435    4282 scope.go:122] "RemoveContainer" containerID="66794b45ffe05dbd78307ae0412993aa70de6d6b435426c439ca788ef3ac090a"
Oct 30 22:00:32.025153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.025122    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"]
Oct 30 22:00:32.033982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.033949    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-4lkwr"]
Oct 30 22:00:32.138647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.138610    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 22:00:32.139533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.139510    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 22:00:32.140901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.140879    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 22:00:32.911335 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.911300    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"]
Oct 30 22:00:32.930606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:32.930528    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"]
Oct 30 22:00:33.482297 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:33.480893    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" path="/var/lib/kubelet/pods/4918fb2b-e3d4-4584-93f0-46cbdad0e656/volumes"
Oct 30 22:00:34.438460 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.438423    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"]
Oct 30 22:00:34.439708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439677    4282 cpu_manager.go:414] "containerMap: removing container" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy"
Oct 30 22:00:34.439708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439704    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439725    4282 cpu_manager.go:414] "containerMap: removing container" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439732    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323ba2ff-6ef1-4923-b4bd-c8c4bc9f680b" containerName="cleanup"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439755    4282 cpu_manager.go:414] "containerMap: removing container" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439762    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439777    4282 cpu_manager.go:414] "containerMap: removing container" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439783    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439797    4282 cpu_manager.go:414] "containerMap: removing container" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439804    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439834    4282 cpu_manager.go:414] "containerMap: removing container" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.439841    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.440049    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.440069    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a21738-5faf-4cea-899c-3be3bc5c62f2" containerName="istio-proxy"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.440091    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="75810534-d64f-4003-89e9-1222cb927848" containerName="istio-proxy"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.440103    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4918fb2b-e3d4-4584-93f0-46cbdad0e656" containerName="istio-proxy"
Oct 30 22:00:34.440305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.440113    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:34.445787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.445768    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.462281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.462255    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"]
Oct 30 22:00:34.465875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.465847    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ecffad-serving-cert\""
Oct 30 22:00:34.481973 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.481943    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.482141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.482013    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72193abe-3cdb-4303-a216-6c3a64e5925b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.482208 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.482147    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftcfc\" (UniqueName: \"kubernetes.io/projected/72193abe-3cdb-4303-a216-6c3a64e5925b-kube-api-access-ftcfc\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.482279 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.482258    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-metrics-certs\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.606340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.598361    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.617490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.616689    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72193abe-3cdb-4303-a216-6c3a64e5925b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.617490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.616824    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftcfc\" (UniqueName: \"kubernetes.io/projected/72193abe-3cdb-4303-a216-6c3a64e5925b-kube-api-access-ftcfc\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.617490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.617000    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-metrics-certs\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.624702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.624669    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72193abe-3cdb-4303-a216-6c3a64e5925b-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.624934 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:34.607584    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-serving-cert: secret "maas-api-e2e-praxis-a-ecffad-serving-cert" not found
Oct 30 22:00:34.625117 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:34.625102    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls podName:72193abe-3cdb-4303-a216-6c3a64e5925b nodeName:}" failed. No retries permitted until 2026-10-30 22:00:35.125078286 +0000 UTC m=+1253.783174499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls") pod "maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv" (UID: "72193abe-3cdb-4303-a216-6c3a64e5925b") : secret "maas-api-e2e-praxis-a-ecffad-serving-cert" not found
Oct 30 22:00:34.657022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.654461    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-metrics-certs\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.657022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.656269    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftcfc\" (UniqueName: \"kubernetes.io/projected/72193abe-3cdb-4303-a216-6c3a64e5925b-kube-api-access-ftcfc\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:34.694748 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:34.694650    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:00:34.770414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:34.770379    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn" podUID="51ce10b9-1687-4796-aa7e-8479ea1e6dde" containerName="payload-pre-processing" containerID="cri-o://8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e" gracePeriod=25
Oct 30 22:00:35.136496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.133939    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 22:00:35.155490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.151054    4282 generic.go:356] "Generic (PLEG): container finished" podID="51ce10b9-1687-4796-aa7e-8479ea1e6dde" containerID="8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e" exitCode=0
Oct 30 22:00:35.155490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.151251    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn" event={"ID":"51ce10b9-1687-4796-aa7e-8479ea1e6dde","Type":"ContainerDied","Data":"8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e"}
Oct 30 22:00:35.155490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.151301    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn" event={"ID":"51ce10b9-1687-4796-aa7e-8479ea1e6dde","Type":"ContainerDied","Data":"61a64c89800298317115da813b5110ac0f81b5ce3dd14d2acce656b0d9429e45"}
Oct 30 22:00:35.155490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.151331    4282 scope.go:122] "RemoveContainer" containerID="8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e"
Oct 30 22:00:35.187492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.177807    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:35.231346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.231263    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:35.304499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.287257    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ce10b9-1687-4796-aa7e-8479ea1e6dde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce10b9-1687-4796-aa7e-8479ea1e6dde-plugins-config-volume\") pod \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\" (UID: \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\") "
Oct 30 22:00:35.304499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.287580    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ce10b9-1687-4796-aa7e-8479ea1e6dde-kube-api-access-hbn49\" (UniqueName: \"kubernetes.io/projected/51ce10b9-1687-4796-aa7e-8479ea1e6dde-kube-api-access-hbn49\") pod \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\" (UID: \"51ce10b9-1687-4796-aa7e-8479ea1e6dde\") "
Oct 30 22:00:35.343965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.343909    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ce10b9-1687-4796-aa7e-8479ea1e6dde-plugins-config-volume" pod "51ce10b9-1687-4796-aa7e-8479ea1e6dde" (UID: "51ce10b9-1687-4796-aa7e-8479ea1e6dde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:35.361401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.360259    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:35.388541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.388415    4282 scope.go:122] "RemoveContainer" containerID="8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e"
Oct 30 22:00:35.388541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.388455    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ce10b9-1687-4796-aa7e-8479ea1e6dde-kube-api-access-hbn49" pod "51ce10b9-1687-4796-aa7e-8479ea1e6dde" (UID: "51ce10b9-1687-4796-aa7e-8479ea1e6dde"). InnerVolumeSpecName "kube-api-access-hbn49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:35.389875 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:35.389847    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e\": container with ID starting with 8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e not found: ID does not exist" containerID="8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e"
Oct 30 22:00:35.389987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.389883    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e"} err="failed to get container status \"8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e\": rpc error: code = NotFound desc = could not find container \"8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e\": container with ID starting with 8e946b58295b2fa99657c2a01d96447b631d36102a40b247c9541e6ca2e2021e not found: ID does not exist"
Oct 30 22:00:35.407739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.407713    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51ce10b9-1687-4796-aa7e-8479ea1e6dde-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:35.407852 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.407746    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbn49\" (UniqueName: \"kubernetes.io/projected/51ce10b9-1687-4796-aa7e-8479ea1e6dde-kube-api-access-hbn49\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:35.760525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:35.760448    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"]
Oct 30 22:00:36.159028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:36.158937    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"
Oct 30 22:00:36.160415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:36.160378    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv" event={"ID":"72193abe-3cdb-4303-a216-6c3a64e5925b","Type":"ContainerStarted","Data":"9be950c8b82d4874f21f6030b4e0814b4dd3ed542c3abc190cd8c8c552426456"}
Oct 30 22:00:36.177585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:36.177563    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"]
Oct 30 22:00:36.183701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:36.183681    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8c6cf1-68bfb95df9-4fltn"]
Oct 30 22:00:37.169534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:37.169499    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv" event={"ID":"72193abe-3cdb-4303-a216-6c3a64e5925b","Type":"ContainerStarted","Data":"f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa"}
Oct 30 22:00:37.169961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:37.169588    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:37.188283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:37.188178    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv" podStartSLOduration=2.715807544 podStartE2EDuration="3.188154587s" totalImagesPullingTime="472.347043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:37.187835929 +0000 UTC m=+1255.845932152" watchObservedRunningTime="2026-10-30 22:00:37.188154587 +0000 UTC m=+1255.846250810"
Oct 30 22:00:37.545567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:37.542583    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ce10b9-1687-4796-aa7e-8479ea1e6dde" path="/var/lib/kubelet/pods/51ce10b9-1687-4796-aa7e-8479ea1e6dde/volumes"
Oct 30 22:00:37.861896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:37.861805    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"]
Oct 30 22:00:37.862122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:37.862092    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy" containerID="cri-o://bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998" gracePeriod=30
Oct 30 22:00:37.912268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:37.912222    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c" podUID="b880fc54-b052-4ba0-a5ea-93ad7aea2419" containerName="payload-processing" containerID="cri-o://7b8719a38570f6c8d60c44f0e9f6b3b23f42430fb2ce2d47188d562ec3010b00" gracePeriod=25
Oct 30 22:00:37.932967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:37.932908    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5" podUID="aea4f226-8ad6-4b30-af4d-094d681c1a6d" containerName="payload-pre-processing" containerID="cri-o://3aa09a182ad0c33d3abfbe5be94b555cf71c7f5fe0eeb6cd125a9cffe09786f5" gracePeriod=25
Oct 30 22:00:38.186954 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.186922    4282 generic.go:356] "Generic (PLEG): container finished" podID="aea4f226-8ad6-4b30-af4d-094d681c1a6d" containerID="3aa09a182ad0c33d3abfbe5be94b555cf71c7f5fe0eeb6cd125a9cffe09786f5" exitCode=0
Oct 30 22:00:38.187403 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.186997    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5" event={"ID":"aea4f226-8ad6-4b30-af4d-094d681c1a6d","Type":"ContainerDied","Data":"3aa09a182ad0c33d3abfbe5be94b555cf71c7f5fe0eeb6cd125a9cffe09786f5"}
Oct 30 22:00:38.189115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.189076    4282 generic.go:356] "Generic (PLEG): container finished" podID="b880fc54-b052-4ba0-a5ea-93ad7aea2419" containerID="7b8719a38570f6c8d60c44f0e9f6b3b23f42430fb2ce2d47188d562ec3010b00" exitCode=0
Oct 30 22:00:38.189337 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.189267    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c" event={"ID":"b880fc54-b052-4ba0-a5ea-93ad7aea2419","Type":"ContainerDied","Data":"7b8719a38570f6c8d60c44f0e9f6b3b23f42430fb2ce2d47188d562ec3010b00"}
Oct 30 22:00:38.220936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.220909    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:38.288436 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.288408    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:38.367521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.367455    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b880fc54-b052-4ba0-a5ea-93ad7aea2419-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b880fc54-b052-4ba0-a5ea-93ad7aea2419-plugins-config-volume\") pod \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\" (UID: \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\") "
Oct 30 22:00:38.367694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.367582    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea4f226-8ad6-4b30-af4d-094d681c1a6d-kube-api-access-qwfxs\" (UniqueName: \"kubernetes.io/projected/aea4f226-8ad6-4b30-af4d-094d681c1a6d-kube-api-access-qwfxs\") pod \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\" (UID: \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\") "
Oct 30 22:00:38.367694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.367646    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea4f226-8ad6-4b30-af4d-094d681c1a6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea4f226-8ad6-4b30-af4d-094d681c1a6d-plugins-config-volume\") pod \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\" (UID: \"aea4f226-8ad6-4b30-af4d-094d681c1a6d\") "
Oct 30 22:00:38.367822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.367778    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b880fc54-b052-4ba0-a5ea-93ad7aea2419-kube-api-access-dnf6m\" (UniqueName: \"kubernetes.io/projected/b880fc54-b052-4ba0-a5ea-93ad7aea2419-kube-api-access-dnf6m\") pod \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\" (UID: \"b880fc54-b052-4ba0-a5ea-93ad7aea2419\") "
Oct 30 22:00:38.367889 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.367843    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b880fc54-b052-4ba0-a5ea-93ad7aea2419-plugins-config-volume" pod "b880fc54-b052-4ba0-a5ea-93ad7aea2419" (UID: "b880fc54-b052-4ba0-a5ea-93ad7aea2419"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:38.368086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.368055    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea4f226-8ad6-4b30-af4d-094d681c1a6d-plugins-config-volume" pod "aea4f226-8ad6-4b30-af4d-094d681c1a6d" (UID: "aea4f226-8ad6-4b30-af4d-094d681c1a6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:38.369098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.369061    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b880fc54-b052-4ba0-a5ea-93ad7aea2419-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:38.369098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.369095    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aea4f226-8ad6-4b30-af4d-094d681c1a6d-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:38.370136 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.370107    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea4f226-8ad6-4b30-af4d-094d681c1a6d-kube-api-access-qwfxs" pod "aea4f226-8ad6-4b30-af4d-094d681c1a6d" (UID: "aea4f226-8ad6-4b30-af4d-094d681c1a6d"). InnerVolumeSpecName "kube-api-access-qwfxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:38.370221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.370150    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b880fc54-b052-4ba0-a5ea-93ad7aea2419-kube-api-access-dnf6m" pod "b880fc54-b052-4ba0-a5ea-93ad7aea2419" (UID: "b880fc54-b052-4ba0-a5ea-93ad7aea2419"). InnerVolumeSpecName "kube-api-access-dnf6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:38.471560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.470308    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwfxs\" (UniqueName: \"kubernetes.io/projected/aea4f226-8ad6-4b30-af4d-094d681c1a6d-kube-api-access-qwfxs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:38.471560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.470342    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnf6m\" (UniqueName: \"kubernetes.io/projected/b880fc54-b052-4ba0-a5ea-93ad7aea2419-kube-api-access-dnf6m\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:38.543313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:38.543287    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:00:39.220559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.219124    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5" event={"ID":"aea4f226-8ad6-4b30-af4d-094d681c1a6d","Type":"ContainerDied","Data":"aa1b15110df64172c1354f4e4571e0ec6a748c9ed29b88e702a6ed346ffb963d"}
Oct 30 22:00:39.220559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.219200    4282 scope.go:122] "RemoveContainer" containerID="3aa09a182ad0c33d3abfbe5be94b555cf71c7f5fe0eeb6cd125a9cffe09786f5"
Oct 30 22:00:39.220559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.219358    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"
Oct 30 22:00:39.248210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.248152    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c" event={"ID":"b880fc54-b052-4ba0-a5ea-93ad7aea2419","Type":"ContainerDied","Data":"3d4e2af9501bdd329fc34fe4db03147ef3e09c0d17eb50c39643e64a97db1296"}
Oct 30 22:00:39.248368 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.248241    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"
Oct 30 22:00:39.252402 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.252382    4282 scope.go:122] "RemoveContainer" containerID="7b8719a38570f6c8d60c44f0e9f6b3b23f42430fb2ce2d47188d562ec3010b00"
Oct 30 22:00:39.278146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.278121    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"]
Oct 30 22:00:39.293298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.293274    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-64f6bdfc75-8mzb5"]
Oct 30 22:00:39.364903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.364881    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"]
Oct 30 22:00:39.384036 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.384014    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-548f47c585-89m6c"]
Oct 30 22:00:39.464521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.464454    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea4f226-8ad6-4b30-af4d-094d681c1a6d" path="/var/lib/kubelet/pods/aea4f226-8ad6-4b30-af4d-094d681c1a6d/volumes"
Oct 30 22:00:39.465135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:39.465118    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b880fc54-b052-4ba0-a5ea-93ad7aea2419" path="/var/lib/kubelet/pods/b880fc54-b052-4ba0-a5ea-93ad7aea2419/volumes"
Oct 30 22:00:40.648154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648126    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"]
Oct 30 22:00:40.648783 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648768    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b880fc54-b052-4ba0-a5ea-93ad7aea2419" containerName="payload-processing"
Oct 30 22:00:40.648844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648784    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b880fc54-b052-4ba0-a5ea-93ad7aea2419" containerName="payload-processing"
Oct 30 22:00:40.648844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648812    4282 cpu_manager.go:414] "containerMap: removing container" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:40.648844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648817    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:40.648943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648847    4282 cpu_manager.go:414] "containerMap: removing container" podUID="aea4f226-8ad6-4b30-af4d-094d681c1a6d" containerName="payload-pre-processing"
Oct 30 22:00:40.648943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648853    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea4f226-8ad6-4b30-af4d-094d681c1a6d" containerName="payload-pre-processing"
Oct 30 22:00:40.648943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648863    4282 cpu_manager.go:414] "containerMap: removing container" podUID="51ce10b9-1687-4796-aa7e-8479ea1e6dde" containerName="payload-pre-processing"
Oct 30 22:00:40.648943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648868    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ce10b9-1687-4796-aa7e-8479ea1e6dde" containerName="payload-pre-processing"
Oct 30 22:00:40.649069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648962    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b880fc54-b052-4ba0-a5ea-93ad7aea2419" containerName="payload-processing"
Oct 30 22:00:40.649069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648975    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b637f3e-c4ba-4fbe-9e94-7121d1df417b" containerName="cleanup"
Oct 30 22:00:40.649069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648985    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea4f226-8ad6-4b30-af4d-094d681c1a6d" containerName="payload-pre-processing"
Oct 30 22:00:40.649069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.648991    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ce10b9-1687-4796-aa7e-8479ea1e6dde" containerName="payload-pre-processing"
Oct 30 22:00:40.653991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.653974    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:40.661113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.661094    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e5289f\""
Oct 30 22:00:40.667276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.667245    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"]
Oct 30 22:00:40.671701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.671684    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"]
Oct 30 22:00:40.684487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.679685    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:40.691837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.691812    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e5289f-dockercfg-9cq4k\""
Oct 30 22:00:40.706085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.706064    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"]
Oct 30 22:00:40.830495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.822154    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpq2q\" (UniqueName: \"kubernetes.io/projected/c95cc705-3ffa-4604-97d5-84f50dc54d2c-kube-api-access-hpq2q\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr\" (UID: \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:40.830495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.822396    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95cc705-3ffa-4604-97d5-84f50dc54d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr\" (UID: \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:40.830495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.822508    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520bdd2e-92e3-4570-aecd-2932ec9b21f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc\" (UID: \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:40.830495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.822607    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8v9\" (UniqueName: \"kubernetes.io/projected/520bdd2e-92e3-4570-aecd-2932ec9b21f4-kube-api-access-jj8v9\") pod \"payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc\" (UID: \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:40.935501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.933724    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpq2q\" (UniqueName: \"kubernetes.io/projected/c95cc705-3ffa-4604-97d5-84f50dc54d2c-kube-api-access-hpq2q\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr\" (UID: \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:40.935501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.933929    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95cc705-3ffa-4604-97d5-84f50dc54d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr\" (UID: \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:40.935501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.933977    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520bdd2e-92e3-4570-aecd-2932ec9b21f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc\" (UID: \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:40.935501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.934053    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8v9\" (UniqueName: \"kubernetes.io/projected/520bdd2e-92e3-4570-aecd-2932ec9b21f4-kube-api-access-jj8v9\") pod \"payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc\" (UID: \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:40.935916 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.935627    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95cc705-3ffa-4604-97d5-84f50dc54d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr\" (UID: \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:40.941487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.936380    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520bdd2e-92e3-4570-aecd-2932ec9b21f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc\" (UID: \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:40.965391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.965358    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8v9\" (UniqueName: \"kubernetes.io/projected/520bdd2e-92e3-4570-aecd-2932ec9b21f4-kube-api-access-jj8v9\") pod \"payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc\" (UID: \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:40.999499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:40.996559    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpq2q\" (UniqueName: \"kubernetes.io/projected/c95cc705-3ffa-4604-97d5-84f50dc54d2c-kube-api-access-hpq2q\") pod \"payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr\" (UID: \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:41.010747 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:41.010716    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:41.289544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:41.288961    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:41.385614 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:41.385590    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"]
Oct 30 22:00:41.388540 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:41.388508    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520bdd2e_92e3_4570_aecd_2932ec9b21f4.slice/crio-b9ce50e57fd672437715230942256b1fc263a657177e521866ecc053badeb791 WatchSource:0}: Error finding container b9ce50e57fd672437715230942256b1fc263a657177e521866ecc053badeb791: Status 404 returned error can't find the container with id b9ce50e57fd672437715230942256b1fc263a657177e521866ecc053badeb791
Oct 30 22:00:41.623128 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:41.623098    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"]
Oct 30 22:00:41.623548 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:41.623525    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95cc705_3ffa_4604_97d5_84f50dc54d2c.slice/crio-f4773ed8f05f6fb6781bcad90d58cf04cc7cc66f50b2213255fc7a8c80273e03 WatchSource:0}: Error finding container f4773ed8f05f6fb6781bcad90d58cf04cc7cc66f50b2213255fc7a8c80273e03: Status 404 returned error can't find the container with id f4773ed8f05f6fb6781bcad90d58cf04cc7cc66f50b2213255fc7a8c80273e03
Oct 30 22:00:42.339502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:42.339426    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc" event={"ID":"520bdd2e-92e3-4570-aecd-2932ec9b21f4","Type":"ContainerStarted","Data":"b9ce50e57fd672437715230942256b1fc263a657177e521866ecc053badeb791"}
Oct 30 22:00:42.341728 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:42.341695    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr" event={"ID":"c95cc705-3ffa-4604-97d5-84f50dc54d2c","Type":"ContainerStarted","Data":"f4773ed8f05f6fb6781bcad90d58cf04cc7cc66f50b2213255fc7a8c80273e03"}
Oct 30 22:00:43.174024 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.173996    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 22:00:43.199063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.199015    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:00:43.319930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.319898    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-token\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-token\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.320138 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.320006    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-socket\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.320660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.320389    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-data\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.320660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.320433    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-podinfo\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.320660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.320505    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istiod-ca-cert\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.320660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.320569    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-credential-socket\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.320660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.320620    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-kube-api-access-pfdfc\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-kube-api-access-pfdfc\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.320999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.320678    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-envoy\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.320999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.320834    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-certs\") pod \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\" (UID: \"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc\") "
Oct 30 22:00:43.321205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.321159    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-data" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:43.321700 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.321681    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.322126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.322108    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-socket" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:43.322276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.322256    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-credential-socket" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:43.322345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.322293    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-certs" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:43.322418 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.322398    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istiod-ca-cert" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:43.324659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.324631    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-token" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:43.324844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.324821    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-envoy" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:43.325025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.325006    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-kube-api-access-pfdfc" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "kube-api-access-pfdfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:43.326574 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.326023    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-podinfo" pod "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" (UID: "17f3ad33-a4b4-4dea-aa9f-e4773f7675cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:43.355025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.354987    4282 generic.go:356] "Generic (PLEG): container finished" podID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerID="bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998" exitCode=0
Oct 30 22:00:43.355387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.355065    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"
Oct 30 22:00:43.355387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.355092    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" event={"ID":"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc","Type":"ContainerDied","Data":"bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998"}
Oct 30 22:00:43.355387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.355131    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn" event={"ID":"17f3ad33-a4b4-4dea-aa9f-e4773f7675cc","Type":"ContainerDied","Data":"c768a3e690cb8fcc13c2882f0e335557fb309b5fe3825c043f00674faccd3566"}
Oct 30 22:00:43.355387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.355157    4282 scope.go:122] "RemoveContainer" containerID="bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998"
Oct 30 22:00:43.412522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.411216    4282 scope.go:122] "RemoveContainer" containerID="bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998"
Oct 30 22:00:43.413026 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.412991    4282 scope.go:122] "RemoveContainer" containerID="bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998"
Oct 30 22:00:43.413641 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:43.413601    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998\": container with ID starting with bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998 not found: ID does not exist" containerID="bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998"
Oct 30 22:00:43.413764 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.413645    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998"} err="failed to get container status \"bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998\": rpc error: code = NotFound desc = could not find container \"bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998\": container with ID starting with bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998 not found: ID does not exist"
Oct 30 22:00:43.423492 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:43.415026    4282 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998\": rpc error: code = NotFound desc = could not find container \"bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998\": container with ID starting with bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998 not found: ID does not exist" containerID="bf7ecc7483dfc7a43152827fe85ef13599d8909e637aac2efc077866a29d9998"
Oct 30 22:00:43.465918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.463157    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.465918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.463193    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.465918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.463210    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.465918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.463224    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.465918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.463238    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfdfc\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-kube-api-access-pfdfc\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.465918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.463252    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.465918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.463265    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.465918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.463278    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:43.500833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.499695    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"]
Oct 30 22:00:43.500833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:43.500346    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-nrvmn"]
Oct 30 22:00:44.364930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:44.364845    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc" event={"ID":"520bdd2e-92e3-4570-aecd-2932ec9b21f4","Type":"ContainerStarted","Data":"fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0"}
Oct 30 22:00:44.365373 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:44.365080    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:44.366935 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:44.366904    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr" event={"ID":"c95cc705-3ffa-4604-97d5-84f50dc54d2c","Type":"ContainerStarted","Data":"6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1"}
Oct 30 22:00:44.367057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:44.366984    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:44.388901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:44.388856    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc" podStartSLOduration=1.813625563 podStartE2EDuration="4.388843178s" totalImagesPullingTime="2.575217615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:44.383637971 +0000 UTC m=+1263.041734193" watchObservedRunningTime="2026-10-30 22:00:44.388843178 +0000 UTC m=+1263.046939401"
Oct 30 22:00:44.403989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:44.403948    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr" podStartSLOduration=2.023842554 podStartE2EDuration="4.403936857s" totalImagesPullingTime="2.380094303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:44.40350746 +0000 UTC m=+1263.061603680" watchObservedRunningTime="2026-10-30 22:00:44.403936857 +0000 UTC m=+1263.062033080"
Oct 30 22:00:45.464952 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:45.464921    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" path="/var/lib/kubelet/pods/17f3ad33-a4b4-4dea-aa9f-e4773f7675cc/volumes"
Oct 30 22:00:47.081019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.080984    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"]
Oct 30 22:00:47.081520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.081299    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn" podUID="f879727f-02e4-4126-af3d-34b4bc97279a" containerName="maas-api" containerID="cri-o://0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a" gracePeriod=30
Oct 30 22:00:47.663778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.663746    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 22:00:47.702919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.702874    4282 generic.go:356] "Generic (PLEG): container finished" podID="f879727f-02e4-4126-af3d-34b4bc97279a" containerID="0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a" exitCode=0
Oct 30 22:00:47.703123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.703102    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn" event={"ID":"f879727f-02e4-4126-af3d-34b4bc97279a","Type":"ContainerDied","Data":"0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a"}
Oct 30 22:00:47.703216 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.703162    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn" event={"ID":"f879727f-02e4-4126-af3d-34b4bc97279a","Type":"ContainerDied","Data":"03ca674da047467e4ba3da037f6092931d0ed4516abcea236f9d6c463a27c351"}
Oct 30 22:00:47.703216 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.703193    4282 scope.go:122] "RemoveContainer" containerID="0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a"
Oct 30 22:00:47.705165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.704811    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"
Oct 30 22:00:47.719499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.719457    4282 scope.go:122] "RemoveContainer" containerID="0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a"
Oct 30 22:00:47.720037 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:47.719994    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a\": container with ID starting with 0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a not found: ID does not exist" containerID="0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a"
Oct 30 22:00:47.720216 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.720189    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a"} err="failed to get container status \"0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a\": rpc error: code = NotFound desc = could not find container \"0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a\": container with ID starting with 0901f70eab09d1b2d4cfcb2dfde8605be6d9199fed7f35c36777bc571048898a not found: ID does not exist"
Oct 30 22:00:47.763345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.763261    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f879727f-02e4-4126-af3d-34b4bc97279a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f879727f-02e4-4126-af3d-34b4bc97279a-openshift-service-ca\") pod \"f879727f-02e4-4126-af3d-34b4bc97279a\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") "
Oct 30 22:00:47.763345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.763314    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f879727f-02e4-4126-af3d-34b4bc97279a-kube-api-access-kwzhh\" (UniqueName: \"kubernetes.io/projected/f879727f-02e4-4126-af3d-34b4bc97279a-kube-api-access-kwzhh\") pod \"f879727f-02e4-4126-af3d-34b4bc97279a\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") "
Oct 30 22:00:47.763601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.763579    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-maas-api-tls\") pod \"f879727f-02e4-4126-af3d-34b4bc97279a\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") "
Oct 30 22:00:47.763713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.763687    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f879727f-02e4-4126-af3d-34b4bc97279a-openshift-service-ca" pod "f879727f-02e4-4126-af3d-34b4bc97279a" (UID: "f879727f-02e4-4126-af3d-34b4bc97279a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:47.763789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.763698    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-metrics-certs\") pod \"f879727f-02e4-4126-af3d-34b4bc97279a\" (UID: \"f879727f-02e4-4126-af3d-34b4bc97279a\") "
Oct 30 22:00:47.764744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.764717    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f879727f-02e4-4126-af3d-34b4bc97279a-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:47.766039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.766011    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-maas-api-tls" pod "f879727f-02e4-4126-af3d-34b4bc97279a" (UID: "f879727f-02e4-4126-af3d-34b4bc97279a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:47.766139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.766084    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f879727f-02e4-4126-af3d-34b4bc97279a-kube-api-access-kwzhh" pod "f879727f-02e4-4126-af3d-34b4bc97279a" (UID: "f879727f-02e4-4126-af3d-34b4bc97279a"). InnerVolumeSpecName "kube-api-access-kwzhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:47.766766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.766748    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-metrics-certs" pod "f879727f-02e4-4126-af3d-34b4bc97279a" (UID: "f879727f-02e4-4126-af3d-34b4bc97279a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:47.872552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.866835    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:47.872552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.866871    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f879727f-02e4-4126-af3d-34b4bc97279a-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:47.872552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:47.866915    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwzhh\" (UniqueName: \"kubernetes.io/projected/f879727f-02e4-4126-af3d-34b4bc97279a-kube-api-access-kwzhh\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:48.030791 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:48.030718    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"]
Oct 30 22:00:48.039110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:48.039075    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8c6cf1-fd4f9d48-jvhhn"]
Oct 30 22:00:48.661858 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:48.661663    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"]
Oct 30 22:00:48.662647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:48.662598    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd" podUID="28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" containerName="maas-api" containerID="cri-o://2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f" gracePeriod=30
Oct 30 22:00:48.984050 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:48.984026    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 22:00:49.083461 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.083430    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-metrics-certs\") pod \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") "
Oct 30 22:00:49.083651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.083532    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-openshift-service-ca\") pod \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") "
Oct 30 22:00:49.083807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.083785    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-maas-api-tls\") pod \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") "
Oct 30 22:00:49.083856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.083846    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-kube-api-access-8jl86\" (UniqueName: \"kubernetes.io/projected/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-kube-api-access-8jl86\") pod \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\" (UID: \"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e\") "
Oct 30 22:00:49.083901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.083857    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-openshift-service-ca" pod "28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" (UID: "28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:49.084681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.084656    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:49.085829 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.085803    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-metrics-certs" pod "28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" (UID: "28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:49.085968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.085950    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-kube-api-access-8jl86" pod "28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" (UID: "28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e"). InnerVolumeSpecName "kube-api-access-8jl86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:49.086044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.085992    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-maas-api-tls" pod "28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" (UID: "28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:49.186298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.186268    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:49.186298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.186293    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:49.186298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.186303    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jl86\" (UniqueName: \"kubernetes.io/projected/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e-kube-api-access-8jl86\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:49.535871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.535834    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f879727f-02e4-4126-af3d-34b4bc97279a" path="/var/lib/kubelet/pods/f879727f-02e4-4126-af3d-34b4bc97279a/volumes"
Oct 30 22:00:49.721813 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.721778    4282 generic.go:356] "Generic (PLEG): container finished" podID="28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" containerID="2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f" exitCode=0
Oct 30 22:00:49.722276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.721861    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"
Oct 30 22:00:49.722276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.721881    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd" event={"ID":"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e","Type":"ContainerDied","Data":"2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f"}
Oct 30 22:00:49.722276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.721942    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd" event={"ID":"28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e","Type":"ContainerDied","Data":"ae12cef140086d32aff7264a3773083a82ca57727a619e0548d86b70393ab37d"}
Oct 30 22:00:49.722276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.721976    4282 scope.go:122] "RemoveContainer" containerID="2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f"
Oct 30 22:00:49.731660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.731644    4282 scope.go:122] "RemoveContainer" containerID="2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f"
Oct 30 22:00:49.731929 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:49.731909    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f\": container with ID starting with 2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f not found: ID does not exist" containerID="2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f"
Oct 30 22:00:49.732021 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.731934    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f"} err="failed to get container status \"2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f\": rpc error: code = NotFound desc = could not find container \"2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f\": container with ID starting with 2302cc0b4ebac461036682bee6ea238a43a74be23c40ee70e22074f69bfbfb4f not found: ID does not exist"
Oct 30 22:00:49.773626 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.773590    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"]
Oct 30 22:00:49.777133 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:49.777111    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0b3fe4-8575d57c5f-j76xd"]
Oct 30 22:00:50.888053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.888021    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"]
Oct 30 22:00:50.888910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.888890    4282 cpu_manager.go:414] "containerMap: removing container" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy"
Oct 30 22:00:50.889032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.888912    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy"
Oct 30 22:00:50.889032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.888949    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f879727f-02e4-4126-af3d-34b4bc97279a" containerName="maas-api"
Oct 30 22:00:50.889032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.888956    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f879727f-02e4-4126-af3d-34b4bc97279a" containerName="maas-api"
Oct 30 22:00:50.889032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.888983    4282 cpu_manager.go:414] "containerMap: removing container" podUID="28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" containerName="maas-api"
Oct 30 22:00:50.889032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.888991    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" containerName="maas-api"
Oct 30 22:00:50.889322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.889137    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f879727f-02e4-4126-af3d-34b4bc97279a" containerName="maas-api"
Oct 30 22:00:50.889322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.889153    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" containerName="maas-api"
Oct 30 22:00:50.889322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.889171    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f3ad33-a4b4-4dea-aa9f-e4773f7675cc" containerName="istio-proxy"
Oct 30 22:00:50.894237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.894216    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.906566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.906548    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3724d9-openshift-default-dockercfg-8zgwq\""
Oct 30 22:00:50.940750 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.940712    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-istio-token\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.940872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.940762    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-data\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.940872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.940788    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-certs\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.940987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.940901    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-envoy\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.941431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.940988    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16347946-2949-44e3-84fd-3621c321ad08-istiod-ca-cert\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.941431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.941017    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrc5\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-kube-api-access-zkrc5\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.941431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.941121    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-credential-socket\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.941431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.941242    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16347946-2949-44e3-84fd-3621c321ad08-istio-podinfo\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.941431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.941280    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-socket\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:50.942661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:50.942639    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"]
Oct 30 22:00:51.048549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.048445    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16347946-2949-44e3-84fd-3621c321ad08-istiod-ca-cert\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.069494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.065603    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrc5\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-kube-api-access-zkrc5\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.069494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.065796    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-credential-socket\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.069494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.065982    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16347946-2949-44e3-84fd-3621c321ad08-istio-podinfo\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.069494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.066033    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-socket\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.069494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.066102    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-istio-token\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.069494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.066144    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-data\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.069494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.066174    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-certs\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.069494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.066208    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-envoy\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.091493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.071290    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-socket\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.133920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.097147    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-certs\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.133920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.053307    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16347946-2949-44e3-84fd-3621c321ad08-istiod-ca-cert\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.133920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.097726    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-credential-socket\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.169787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.166886    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-data\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.207783 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.207747    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrc5\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-kube-api-access-zkrc5\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.222494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.219319    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16347946-2949-44e3-84fd-3621c321ad08-istio-podinfo\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.255599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.255560    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-envoy\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.269769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.264003    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-istio-token\") pod \"e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") " pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.469135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.469054    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e" path="/var/lib/kubelet/pods/28fb0fdb-6afb-4b25-ab9f-b98b2a2eb67e/volumes"
Oct 30 22:00:51.508192 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.508160    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:51.916402 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.916373    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"]
Oct 30 22:00:51.917600 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:51.917571    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16347946_2949_44e3_84fd_3621c321ad08.slice/crio-9a0e9c3e458d488d6670bc4c22fbbf677a2a9b24f18dc35b63a95db4fcbc63ee WatchSource:0}: Error finding container 9a0e9c3e458d488d6670bc4c22fbbf677a2a9b24f18dc35b63a95db4fcbc63ee: Status 404 returned error can't find the container with id 9a0e9c3e458d488d6670bc4c22fbbf677a2a9b24f18dc35b63a95db4fcbc63ee
Oct 30 22:00:51.920335 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.920302    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:51.920422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.920368    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:51.920422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:51.920401    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:00:52.744795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:52.744762    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" event={"ID":"16347946-2949-44e3-84fd-3621c321ad08","Type":"ContainerStarted","Data":"123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d"}
Oct 30 22:00:52.744988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:52.744813    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" event={"ID":"16347946-2949-44e3-84fd-3621c321ad08","Type":"ContainerStarted","Data":"9a0e9c3e458d488d6670bc4c22fbbf677a2a9b24f18dc35b63a95db4fcbc63ee"}
Oct 30 22:00:52.771676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:52.771624    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" podStartSLOduration=2.771610343 podStartE2EDuration="2.771610343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:52.769836012 +0000 UTC m=+1271.427932235" watchObservedRunningTime="2026-10-30 22:00:52.771610343 +0000 UTC m=+1271.429706566"
Oct 30 22:00:53.166181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:53.166080    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"]
Oct 30 22:00:53.166732 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:53.166283    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy" containerID="cri-o://51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456" gracePeriod=30
Oct 30 22:00:53.508567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:53.508529    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:53.514361 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:53.514337    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:53.830196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:53.830118    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:53.832794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:53.832771    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:00:55.376123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:55.376094    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:00:55.376643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:55.376174    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:00:56.540648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.540617    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"]
Oct 30 22:00:56.550165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.546561    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:56.562440 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.562419    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-3724d9\""
Oct 30 22:00:56.564341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.564321    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-3724d9-dockercfg-mmf8m\""
Oct 30 22:00:56.597575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.597548    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"]
Oct 30 22:00:56.603069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.603048    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"]
Oct 30 22:00:56.603180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.603133    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:56.622291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.622265    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk\" (UID: \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:56.622678 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.622656    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44jd\" (UniqueName: \"kubernetes.io/projected/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-kube-api-access-z44jd\") pod \"payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk\" (UID: \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:56.623657 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.623634    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"]
Oct 30 22:00:56.730623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.730578    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9045806c-c579-4ffe-838a-f31d6960b166-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn\" (UID: \"9045806c-c579-4ffe-838a-f31d6960b166\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:56.741491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.737879    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzjb\" (UniqueName: \"kubernetes.io/projected/9045806c-c579-4ffe-838a-f31d6960b166-kube-api-access-hhzjb\") pod \"payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn\" (UID: \"9045806c-c579-4ffe-838a-f31d6960b166\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:56.769183 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.769144    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk\" (UID: \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:56.769658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.769616    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44jd\" (UniqueName: \"kubernetes.io/projected/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-kube-api-access-z44jd\") pod \"payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk\" (UID: \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:56.807542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.791604    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk\" (UID: \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:56.917566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.917171    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44jd\" (UniqueName: \"kubernetes.io/projected/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-kube-api-access-z44jd\") pod \"payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk\" (UID: \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:56.924217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.923947    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:56.934927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.934897    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9045806c-c579-4ffe-838a-f31d6960b166-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn\" (UID: \"9045806c-c579-4ffe-838a-f31d6960b166\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:56.935454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.935415    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzjb\" (UniqueName: \"kubernetes.io/projected/9045806c-c579-4ffe-838a-f31d6960b166-kube-api-access-hhzjb\") pod \"payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn\" (UID: \"9045806c-c579-4ffe-838a-f31d6960b166\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:56.951548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:56.940434    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9045806c-c579-4ffe-838a-f31d6960b166-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn\" (UID: \"9045806c-c579-4ffe-838a-f31d6960b166\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:57.024349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.022543    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzjb\" (UniqueName: \"kubernetes.io/projected/9045806c-c579-4ffe-838a-f31d6960b166-kube-api-access-hhzjb\") pod \"payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn\" (UID: \"9045806c-c579-4ffe-838a-f31d6960b166\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:57.235503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.234105    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:57.312824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.312798    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"]
Oct 30 22:00:57.313512 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:57.313462    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d07ff68_ceb1_426a_9feb_d3d5ec90029d.slice/crio-0b5ad4f15928c6d2a49f576d6c4a29de0c757e013f5a6364d8e5d0f1eefc5b1d WatchSource:0}: Error finding container 0b5ad4f15928c6d2a49f576d6c4a29de0c757e013f5a6364d8e5d0f1eefc5b1d: Status 404 returned error can't find the container with id 0b5ad4f15928c6d2a49f576d6c4a29de0c757e013f5a6364d8e5d0f1eefc5b1d
Oct 30 22:00:57.561901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.561873    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"]
Oct 30 22:00:57.563104 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:00:57.563077    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9045806c_c579_4ffe_838a_f31d6960b166.slice/crio-38f049d73d1c911c9ecfd0d6f9f530ac35ea7cba328725a582481f9b5b70e1d8 WatchSource:0}: Error finding container 38f049d73d1c911c9ecfd0d6f9f530ac35ea7cba328725a582481f9b5b70e1d8: Status 404 returned error can't find the container with id 38f049d73d1c911c9ecfd0d6f9f530ac35ea7cba328725a582481f9b5b70e1d8
Oct 30 22:00:57.987572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.986733    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk" event={"ID":"9d07ff68-ceb1-426a-9feb-d3d5ec90029d","Type":"ContainerStarted","Data":"5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423"}
Oct 30 22:00:57.987770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.987622    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:00:57.987770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.987649    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk" event={"ID":"9d07ff68-ceb1-426a-9feb-d3d5ec90029d","Type":"ContainerStarted","Data":"0b5ad4f15928c6d2a49f576d6c4a29de0c757e013f5a6364d8e5d0f1eefc5b1d"}
Oct 30 22:00:57.989017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.988988    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn" event={"ID":"9045806c-c579-4ffe-838a-f31d6960b166","Type":"ContainerStarted","Data":"0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd"}
Oct 30 22:00:57.989390 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.989035    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn" event={"ID":"9045806c-c579-4ffe-838a-f31d6960b166","Type":"ContainerStarted","Data":"38f049d73d1c911c9ecfd0d6f9f530ac35ea7cba328725a582481f9b5b70e1d8"}
Oct 30 22:00:57.989390 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:57.989091    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:00:58.011968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.011921    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk" podStartSLOduration=2.011907408 podStartE2EDuration="2.011907408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:58.008992265 +0000 UTC m=+1276.667088487" watchObservedRunningTime="2026-10-30 22:00:58.011907408 +0000 UTC m=+1276.670003630"
Oct 30 22:00:58.029770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.029729    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn" podStartSLOduration=2.0297170429999998 podStartE2EDuration="2.029717043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:58.025949811 +0000 UTC m=+1276.684046034" watchObservedRunningTime="2026-10-30 22:00:58.029717043 +0000 UTC m=+1276.687813265"
Oct 30 22:00:58.300482 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.300433    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:00:58.419633 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.419594    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"]
Oct 30 22:00:58.419873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.419845    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy" containerID="cri-o://7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d" gracePeriod=30
Oct 30 22:00:58.469071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.469048    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585030    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-certs\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585169    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-podinfo\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585218    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istiod-ca-cert\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585257    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-token\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-token\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585311    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-credential-socket\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585374    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-socket\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585538    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-data\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585588    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-kube-api-access-cgknc\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-kube-api-access-cgknc\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.589587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.585770    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-envoy\") pod \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\" (UID: \"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d\") "
Oct 30 22:00:58.612665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.612621    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-socket" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:58.615420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.615386    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-credential-socket" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:58.639492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.627251    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.656565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.640593    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.687997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.685047    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-certs" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:58.693621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.693585    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-data" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:58.710164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.700034    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istiod-ca-cert" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:58.744490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.729910    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-podinfo" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:58.764529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.755418    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.764529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.755458    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.764529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.755491    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.764529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.755506    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.826600 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.816086    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-token" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:58.829046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.827896    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-kube-api-access-cgknc" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "kube-api-access-cgknc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:58.838053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.838019    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-envoy" pod "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" (UID: "8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:58.857029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.856954    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgknc\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-kube-api-access-cgknc\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.857029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.856980    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.857029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.856988    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:58.997095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.997067    4282 generic.go:356] "Generic (PLEG): container finished" podID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerID="51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456" exitCode=0
Oct 30 22:00:58.997267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.997120    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"
Oct 30 22:00:58.997267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.997154    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" event={"ID":"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d","Type":"ContainerDied","Data":"51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456"}
Oct 30 22:00:58.997267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.997210    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm" event={"ID":"8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d","Type":"ContainerDied","Data":"6f79ca46d239ed4fc9ba436eee07986566aa7019b8dd83d30bc78128628e2f59"}
Oct 30 22:00:58.997267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:58.997235    4282 scope.go:122] "RemoveContainer" containerID="51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456"
Oct 30 22:00:59.008147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:59.008129    4282 scope.go:122] "RemoveContainer" containerID="51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456"
Oct 30 22:00:59.008430 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:00:59.008407    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456\": container with ID starting with 51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456 not found: ID does not exist" containerID="51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456"
Oct 30 22:00:59.008553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:59.008438    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456"} err="failed to get container status \"51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456\": rpc error: code = NotFound desc = could not find container \"51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456\": container with ID starting with 51feb3c74b505a22d90b10c459c77012c06cfb2f07d36aca970e88cb8d7b8456 not found: ID does not exist"
Oct 30 22:00:59.023211 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:59.023190    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"]
Oct 30 22:00:59.028978 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:59.028949    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-d2shm"]
Oct 30 22:00:59.542648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:00:59.536763    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" path="/var/lib/kubelet/pods/8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d/volumes"
Oct 30 22:01:01.988677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:01.988646    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"]
Oct 30 22:01:01.989736 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:01.989713    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy"
Oct 30 22:01:01.989736 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:01.989736    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy"
Oct 30 22:01:01.989907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:01.989892    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bab4ca7-c5ce-4dc2-9d72-d1d771e7455d" containerName="istio-proxy"
Oct 30 22:01:01.994435 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:01.994406    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.001961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.001938    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-71f39e-openshift-default-dockercfg-bzsk6\""
Oct 30 22:01:02.012528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.012507    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"]
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.117773    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.117982    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.118159    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.118187    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.118260    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.118289    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.118426    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.118501    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.119411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.118550    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdf8c\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-kube-api-access-wdf8c\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.220549    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.220630    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.220680    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdf8c\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-kube-api-access-wdf8c\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.220803    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.220878    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.220974    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.220998    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.221070    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.221101    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.221654    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.222498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.222282    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.223205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.222550    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.247211 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.233250    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.247211 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.235894    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.247211 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.243988    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.254497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.247884    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.254497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.253112    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdf8c\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-kube-api-access-wdf8c\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.282492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.259488    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.322638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.313790    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:02.756515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.753533    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"]
Oct 30 22:01:02.758121 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:01:02.758092    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2f09f7_ffd1_4f55_82a6_5c7b8f90c78d.slice/crio-8a8e41df50f9a161319c593f2c9533dbd74b793749bc310bd211082b2260de9b WatchSource:0}: Error finding container 8a8e41df50f9a161319c593f2c9533dbd74b793749bc310bd211082b2260de9b: Status 404 returned error can't find the container with id 8a8e41df50f9a161319c593f2c9533dbd74b793749bc310bd211082b2260de9b
Oct 30 22:01:02.762245 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.762198    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:02.763533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.763504    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:02.763716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:02.763694    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:03.067377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.067277    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" event={"ID":"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d","Type":"ContainerStarted","Data":"8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2"}
Oct 30 22:01:03.067377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.067355    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" event={"ID":"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d","Type":"ContainerStarted","Data":"8a8e41df50f9a161319c593f2c9533dbd74b793749bc310bd211082b2260de9b"}
Oct 30 22:01:03.092427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.092381    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" podStartSLOduration=2.092365579 podStartE2EDuration="2.092365579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:03.089169508 +0000 UTC m=+1281.747265731" watchObservedRunningTime="2026-10-30 22:01:03.092365579 +0000 UTC m=+1281.750461803"
Oct 30 22:01:03.326545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.323955    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:03.330842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.330814    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:03.711252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.711229    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 22:01:03.758681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.758647    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-socket\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.758842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.758780    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-istio-token\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-istio-token\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.758921 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.758845    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-envoy\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.758982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.758948    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc999841-a85f-49d0-942e-7a09f76df056-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc999841-a85f-49d0-942e-7a09f76df056-istio-podinfo\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.758982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.758962    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-socket" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:03.759088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.758982    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-certs\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.759193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.759173    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-kube-api-access-pkmh8\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-kube-api-access-pkmh8\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.759267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.759214    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc999841-a85f-49d0-942e-7a09f76df056-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc999841-a85f-49d0-942e-7a09f76df056-istiod-ca-cert\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.759267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.759246    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-data\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.759374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.759301    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-credential-socket\") pod \"dc999841-a85f-49d0-942e-7a09f76df056\" (UID: \"dc999841-a85f-49d0-942e-7a09f76df056\") "
Oct 30 22:01:03.759756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.759685    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-certs" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:03.760504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.760086    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-data" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:03.760504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.760128    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc999841-a85f-49d0-942e-7a09f76df056-istiod-ca-cert" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:03.760504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.760403    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:03.760504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.760427    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:03.760504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.760441    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc999841-a85f-49d0-942e-7a09f76df056-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:03.760504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.760455    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:03.760975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.760935    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-credential-socket" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:03.761848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.761815    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc999841-a85f-49d0-942e-7a09f76df056-istio-podinfo" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:01:03.762234 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.762212    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-kube-api-access-pkmh8" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "kube-api-access-pkmh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:03.762340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.762320    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-envoy" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:03.763927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.763906    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-istio-token" pod "dc999841-a85f-49d0-942e-7a09f76df056" (UID: "dc999841-a85f-49d0-942e-7a09f76df056"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:03.862330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.862257    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc999841-a85f-49d0-942e-7a09f76df056-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:03.862330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.862287    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkmh8\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-kube-api-access-pkmh8\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:03.862330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.862302    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:03.862330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.862326    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc999841-a85f-49d0-942e-7a09f76df056-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:03.862568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:03.862340    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc999841-a85f-49d0-942e-7a09f76df056-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:04.075314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.075279    4282 generic.go:356] "Generic (PLEG): container finished" podID="dc999841-a85f-49d0-942e-7a09f76df056" containerID="7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d" exitCode=0
Oct 30 22:01:04.075820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.075327    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"
Oct 30 22:01:04.075820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.075364    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" event={"ID":"dc999841-a85f-49d0-942e-7a09f76df056","Type":"ContainerDied","Data":"7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d"}
Oct 30 22:01:04.075820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.075417    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq" event={"ID":"dc999841-a85f-49d0-942e-7a09f76df056","Type":"ContainerDied","Data":"e7e3597f7f6d658934c1d568746b6a5201130cb0d405334bb4d47b5ced345c72"}
Oct 30 22:01:04.075820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.075441    4282 scope.go:122] "RemoveContainer" containerID="7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d"
Oct 30 22:01:04.076043 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.075821    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:04.076882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.076861    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:01:04.086254 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.086237    4282 scope.go:122] "RemoveContainer" containerID="d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528"
Oct 30 22:01:04.096324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.096310    4282 scope.go:122] "RemoveContainer" containerID="7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d"
Oct 30 22:01:04.096593 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:04.096576    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d\": container with ID starting with 7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d not found: ID does not exist" containerID="7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d"
Oct 30 22:01:04.096653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.096601    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d"} err="failed to get container status \"7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d\": rpc error: code = NotFound desc = could not find container \"7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d\": container with ID starting with 7f956746c61c6e64e24c5b685fe7b71b3c6ca1dd25908063ebcf31723147101d not found: ID does not exist"
Oct 30 22:01:04.096653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.096616    4282 scope.go:122] "RemoveContainer" containerID="d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528"
Oct 30 22:01:04.096838 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:04.096822    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528\": container with ID starting with d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528 not found: ID does not exist" containerID="d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528"
Oct 30 22:01:04.096875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.096843    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528"} err="failed to get container status \"d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528\": rpc error: code = NotFound desc = could not find container \"d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528\": container with ID starting with d6bc73d6965e83e88a585e86b6a4dff1b45b01853f6581247dfd419a6c09f528 not found: ID does not exist"
Oct 30 22:01:04.213521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.194587    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"]
Oct 30 22:01:04.213521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:04.212449    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8c6cf1-openshift-default-b9bfd447c-f8qsq"]
Oct 30 22:01:04.649664 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:04.649580    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:01:05.465931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:05.465898    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc999841-a85f-49d0-942e-7a09f76df056" path="/var/lib/kubelet/pods/dc999841-a85f-49d0-942e-7a09f76df056/volumes"
Oct 30 22:01:06.818410 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.818378    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"]
Oct 30 22:01:06.819031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.819018    4282 cpu_manager.go:414] "containerMap: removing container" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy"
Oct 30 22:01:06.819076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.819033    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy"
Oct 30 22:01:06.819679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.819653    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy"
Oct 30 22:01:06.819806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.819688    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy"
Oct 30 22:01:06.827139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.827112    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:06.836187 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.836160    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"]
Oct 30 22:01:06.840376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.839740    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-71f39e-dockercfg-r8m7b\""
Oct 30 22:01:06.840376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.839839    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-71f39e\""
Oct 30 22:01:06.852022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.851988    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"]
Oct 30 22:01:06.853027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.853003    4282 cpu_manager.go:414] "containerMap: removing container" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy"
Oct 30 22:01:06.853027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.853028    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc999841-a85f-49d0-942e-7a09f76df056" containerName="istio-proxy"
Oct 30 22:01:06.857483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.857444    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:06.876825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:06.876797    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"]
Oct 30 22:01:07.089551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.086841    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2445cdba-b53a-4f2f-898c-7c0b837e8571-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh\" (UID: \"2445cdba-b53a-4f2f-898c-7c0b837e8571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:07.089551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.087072    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcjdj\" (UniqueName: \"kubernetes.io/projected/174221eb-a62d-49a2-8ffc-e209628e641c-kube-api-access-vcjdj\") pod \"payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9\" (UID: \"174221eb-a62d-49a2-8ffc-e209628e641c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:07.089551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.087149    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgrx\" (UniqueName: \"kubernetes.io/projected/2445cdba-b53a-4f2f-898c-7c0b837e8571-kube-api-access-wtgrx\") pod \"payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh\" (UID: \"2445cdba-b53a-4f2f-898c-7c0b837e8571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:07.089551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.087211    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174221eb-a62d-49a2-8ffc-e209628e641c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9\" (UID: \"174221eb-a62d-49a2-8ffc-e209628e641c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:07.119113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.117555    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"]
Oct 30 22:01:07.172877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.171020    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"]
Oct 30 22:01:07.172877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.171147    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.181538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.176877    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3724d9-serving-cert\""
Oct 30 22:01:07.189132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.188902    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcjdj\" (UniqueName: \"kubernetes.io/projected/174221eb-a62d-49a2-8ffc-e209628e641c-kube-api-access-vcjdj\") pod \"payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9\" (UID: \"174221eb-a62d-49a2-8ffc-e209628e641c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:07.189132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.189009    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgrx\" (UniqueName: \"kubernetes.io/projected/2445cdba-b53a-4f2f-898c-7c0b837e8571-kube-api-access-wtgrx\") pod \"payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh\" (UID: \"2445cdba-b53a-4f2f-898c-7c0b837e8571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:07.189132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.189091    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174221eb-a62d-49a2-8ffc-e209628e641c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9\" (UID: \"174221eb-a62d-49a2-8ffc-e209628e641c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:07.189357 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.189187    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2445cdba-b53a-4f2f-898c-7c0b837e8571-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh\" (UID: \"2445cdba-b53a-4f2f-898c-7c0b837e8571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:07.196636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.190212    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2445cdba-b53a-4f2f-898c-7c0b837e8571-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh\" (UID: \"2445cdba-b53a-4f2f-898c-7c0b837e8571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:07.198698 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.198672    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174221eb-a62d-49a2-8ffc-e209628e641c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9\" (UID: \"174221eb-a62d-49a2-8ffc-e209628e641c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:07.271908 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.271873    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgrx\" (UniqueName: \"kubernetes.io/projected/2445cdba-b53a-4f2f-898c-7c0b837e8571-kube-api-access-wtgrx\") pod \"payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh\" (UID: \"2445cdba-b53a-4f2f-898c-7c0b837e8571\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:07.292427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.292387    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.292621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.292499    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449h5\" (UniqueName: \"kubernetes.io/projected/c0f801ad-9939-400d-b564-c013416f3f6e-kube-api-access-449h5\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.292621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.292550    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-metrics-certs\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.292759 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.292742    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f801ad-9939-400d-b564-c013416f3f6e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.297867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.297837    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcjdj\" (UniqueName: \"kubernetes.io/projected/174221eb-a62d-49a2-8ffc-e209628e641c-kube-api-access-vcjdj\") pod \"payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9\" (UID: \"174221eb-a62d-49a2-8ffc-e209628e641c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:07.399549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.395903    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.399549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.395989    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449h5\" (UniqueName: \"kubernetes.io/projected/c0f801ad-9939-400d-b564-c013416f3f6e-kube-api-access-449h5\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.399549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.396033    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-metrics-certs\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.399549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.396169    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f801ad-9939-400d-b564-c013416f3f6e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.399549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.397053    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f801ad-9939-400d-b564-c013416f3f6e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.399549 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:07.397170    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-serving-cert: secret "maas-api-e2e-praxis-b-3724d9-serving-cert" not found
Oct 30 22:01:07.399549 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:07.397226    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls podName:c0f801ad-9939-400d-b564-c013416f3f6e nodeName:}" failed. No retries permitted until 2026-10-30 22:01:07.897207782 +0000 UTC m=+1286.555304001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls") pod "maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb" (UID: "c0f801ad-9939-400d-b564-c013416f3f6e") : secret "maas-api-e2e-praxis-b-3724d9-serving-cert" not found
Oct 30 22:01:07.442500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.438073    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-449h5\" (UniqueName: \"kubernetes.io/projected/c0f801ad-9939-400d-b564-c013416f3f6e-kube-api-access-449h5\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.461794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.461752    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:07.463790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.463752    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-metrics-certs\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.478748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.478721    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:07.959488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.958344    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:07.969575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:07.962558    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:08.010833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.001961    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"]
Oct 30 22:01:08.037527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.030189    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"]
Oct 30 22:01:08.210644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.210570    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:08.222987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.222939    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh" event={"ID":"2445cdba-b53a-4f2f-898c-7c0b837e8571","Type":"ContainerStarted","Data":"c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c"}
Oct 30 22:01:08.223147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.223006    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh" event={"ID":"2445cdba-b53a-4f2f-898c-7c0b837e8571","Type":"ContainerStarted","Data":"1f1bed1cbfcbae095756d3e3e35829b28c6f992a0cf4f8ee4b3e1f0edb20eb55"}
Oct 30 22:01:08.223212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.223145    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:08.225086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.225065    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9" event={"ID":"174221eb-a62d-49a2-8ffc-e209628e641c","Type":"ContainerStarted","Data":"7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0"}
Oct 30 22:01:08.225205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.225103    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9" event={"ID":"174221eb-a62d-49a2-8ffc-e209628e641c","Type":"ContainerStarted","Data":"d30e6988421319f797098934e037991a2526c34cc56ee7fef215432a0f6ad688"}
Oct 30 22:01:08.225263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.225202    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:08.244848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.244787    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh" podStartSLOduration=2.244773614 podStartE2EDuration="2.244773614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:08.241540862 +0000 UTC m=+1286.899637086" watchObservedRunningTime="2026-10-30 22:01:08.244773614 +0000 UTC m=+1286.902869838"
Oct 30 22:01:08.264651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.264600    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9" podStartSLOduration=2.264586443 podStartE2EDuration="2.264586443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:08.259700185 +0000 UTC m=+1286.917796407" watchObservedRunningTime="2026-10-30 22:01:08.264586443 +0000 UTC m=+1286.922682666"
Oct 30 22:01:08.524328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:08.524299    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"]
Oct 30 22:01:08.525151 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:01:08.525126    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f801ad_9939_400d_b564_c013416f3f6e.slice/crio-63f77dddb19107327fc897e8d3b589930b05d8d3b0aa15269edc91e892130c1c WatchSource:0}: Error finding container 63f77dddb19107327fc897e8d3b589930b05d8d3b0aa15269edc91e892130c1c: Status 404 returned error can't find the container with id 63f77dddb19107327fc897e8d3b589930b05d8d3b0aa15269edc91e892130c1c
Oct 30 22:01:09.011067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:09.009335    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:01:09.020035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:09.020009    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:01:09.233832 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:09.233796    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb" event={"ID":"c0f801ad-9939-400d-b564-c013416f3f6e","Type":"ContainerStarted","Data":"0389e75aadd85931dbda4bc7de0cd8f11e2c08de71798f46bb78b4a9ebbab6d2"}
Oct 30 22:01:09.234046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:09.233851    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:09.234046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:09.233867    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb" event={"ID":"c0f801ad-9939-400d-b564-c013416f3f6e","Type":"ContainerStarted","Data":"63f77dddb19107327fc897e8d3b589930b05d8d3b0aa15269edc91e892130c1c"}
Oct 30 22:01:09.263268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:09.263210    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb" podStartSLOduration=2.877714776 podStartE2EDuration="3.263192471s" totalImagesPullingTime="385.477695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:09.261345893 +0000 UTC m=+1287.919442116" watchObservedRunningTime="2026-10-30 22:01:09.263192471 +0000 UTC m=+1287.921288694"
Oct 30 22:01:15.259228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:15.259198    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:01:16.042307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.042276    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"]
Oct 30 22:01:16.048003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.047888    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.060507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.059640    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"]
Oct 30 22:01:16.064750 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.064730    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-71f39e-serving-cert\""
Oct 30 22:01:16.096175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.096140    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h65nd\" (UniqueName: \"kubernetes.io/projected/24c40bd3-43dd-4005-9d44-d554f060db78-kube-api-access-h65nd\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.096336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.096193    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-maas-api-tls\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.096336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.096311    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-metrics-certs\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.096498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.096348    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c40bd3-43dd-4005-9d44-d554f060db78-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.226946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.220057    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-maas-api-tls\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.226946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.220152    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-metrics-certs\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.226946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.220189    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c40bd3-43dd-4005-9d44-d554f060db78-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.226946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.220782    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h65nd\" (UniqueName: \"kubernetes.io/projected/24c40bd3-43dd-4005-9d44-d554f060db78-kube-api-access-h65nd\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.231218 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.228986    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c40bd3-43dd-4005-9d44-d554f060db78-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.255610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.254601    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-maas-api-tls\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.259913 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.257292    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-metrics-certs\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.259913 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.258607    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h65nd\" (UniqueName: \"kubernetes.io/projected/24c40bd3-43dd-4005-9d44-d554f060db78-kube-api-access-h65nd\") pod \"maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.367570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.367458    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:16.753829 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:01:16.737950    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c40bd3_43dd_4005_9d44_d554f060db78.slice/crio-478b3f10bb6c87a0d4a7d6a08e3403db75ee2b7b44e929f07a5020bd14c0b4a6 WatchSource:0}: Error finding container 478b3f10bb6c87a0d4a7d6a08e3403db75ee2b7b44e929f07a5020bd14c0b4a6: Status 404 returned error can't find the container with id 478b3f10bb6c87a0d4a7d6a08e3403db75ee2b7b44e929f07a5020bd14c0b4a6
Oct 30 22:01:16.793306 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.775327    4282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:01:16.806081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:16.806048    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"]
Oct 30 22:01:17.317483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:17.317443    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc" event={"ID":"24c40bd3-43dd-4005-9d44-d554f060db78","Type":"ContainerStarted","Data":"478b3f10bb6c87a0d4a7d6a08e3403db75ee2b7b44e929f07a5020bd14c0b4a6"}
Oct 30 22:01:18.327295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:18.327254    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc" event={"ID":"24c40bd3-43dd-4005-9d44-d554f060db78","Type":"ContainerStarted","Data":"6a9cea6eb1a4e3c4c3be994e89ecb78445b7f6aed0840dabaf67cf8c19027756"}
Oct 30 22:01:18.327774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:18.327448    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:18.349108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:18.349061    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc" podStartSLOduration=1.879367914 podStartE2EDuration="2.349049721s" totalImagesPullingTime="469.681807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:18.346277137 +0000 UTC m=+1297.004373360" watchObservedRunningTime="2026-10-30 22:01:18.349049721 +0000 UTC m=+1297.007145944"
Oct 30 22:01:19.238859 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:19.238817    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:01:19.239765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:19.238908    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:01:20.998610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:20.998578    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"]
Oct 30 22:01:20.999112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:20.998848    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy" containerID="cri-o://2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6" gracePeriod=30
Oct 30 22:01:24.343284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:24.343213    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:01:25.368649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:25.368616    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:01:26.496704 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.496678    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 22:01:26.497147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.496721    4282 generic.go:356] "Generic (PLEG): container finished" podID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerID="2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6" exitCode=0
Oct 30 22:01:26.497147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.496794    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" event={"ID":"7646623d-2c6a-4b60-90f8-31b67cb69c05","Type":"ContainerDied","Data":"2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6"}
Oct 30 22:01:26.497147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.496872    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd" event={"ID":"7646623d-2c6a-4b60-90f8-31b67cb69c05","Type":"ContainerDied","Data":"833621a4476649d516e7b10c1247498cad225679883ce6b652abc707f316ffcc"}
Oct 30 22:01:26.497147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.496900    4282 scope.go:122] "RemoveContainer" containerID="2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6"
Oct 30 22:01:26.507561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.507544    4282 scope.go:122] "RemoveContainer" containerID="cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e"
Oct 30 22:01:26.519962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.519908    4282 scope.go:122] "RemoveContainer" containerID="2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6"
Oct 30 22:01:26.520335 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:26.520300    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6\": container with ID starting with 2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6 not found: ID does not exist" containerID="2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6"
Oct 30 22:01:26.520442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.520347    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6"} err="failed to get container status \"2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6\": rpc error: code = NotFound desc = could not find container \"2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6\": container with ID starting with 2b8643ac74ad88a6f38657fbcd528efea1553e71d5111b82b6399f4d51057bb6 not found: ID does not exist"
Oct 30 22:01:26.520442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.520367    4282 scope.go:122] "RemoveContainer" containerID="cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e"
Oct 30 22:01:26.520738 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:26.520715    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e\": container with ID starting with cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e not found: ID does not exist" containerID="cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e"
Oct 30 22:01:26.520788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.520751    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e"} err="failed to get container status \"cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e\": rpc error: code = NotFound desc = could not find container \"cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e\": container with ID starting with cbc21844c1c74045912b2adc08bdf1c5967c4a779f67fb549a2e7dc7068b899e not found: ID does not exist"
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.624447    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-podinfo\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.624587    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-socket\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.624622    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-credential-socket\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.624696    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-certs\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.624725    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-data\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.624899    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-token\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-token\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.624955    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-envoy\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.625053    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-kube-api-access-2gzbt\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-kube-api-access-2gzbt\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.626504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.625111    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7646623d-2c6a-4b60-90f8-31b67cb69c05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7646623d-2c6a-4b60-90f8-31b67cb69c05-istiod-ca-cert\") pod \"7646623d-2c6a-4b60-90f8-31b67cb69c05\" (UID: \"7646623d-2c6a-4b60-90f8-31b67cb69c05\") "
Oct 30 22:01:26.726762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.705587    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7646623d-2c6a-4b60-90f8-31b67cb69c05-istiod-ca-cert" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:26.726762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.725587    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-data" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:26.733507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.727608    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-socket" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:26.770567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.736559    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-credential-socket" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:26.770567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.745549    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-certs" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:26.770567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.759315    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7646623d-2c6a-4b60-90f8-31b67cb69c05-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:26.770567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.759347    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:26.770567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.759362    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:26.770567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.759377    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:26.770567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.759390    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:26.901552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.832894    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-kube-api-access-2gzbt" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "kube-api-access-2gzbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:26.901552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.861262    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gzbt\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-kube-api-access-2gzbt\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:26.901552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.895375    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-token" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:26.902576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.902516    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-envoy" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:26.930881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.930840    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-podinfo" pod "7646623d-2c6a-4b60-90f8-31b67cb69c05" (UID: "7646623d-2c6a-4b60-90f8-31b67cb69c05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:01:26.971499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.971100    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:26.971499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.971139    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:26.971499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:26.971155    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7646623d-2c6a-4b60-90f8-31b67cb69c05-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:27.504394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:27.504367    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"
Oct 30 22:01:27.637707 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:27.637634    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"]
Oct 30 22:01:27.798556 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:27.770385    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-19cab1-openshift-default-8d845478c-ch2vd"]
Oct 30 22:01:29.484414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:29.484381    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" path="/var/lib/kubelet/pods/7646623d-2c6a-4b60-90f8-31b67cb69c05/volumes"
Oct 30 22:01:29.803829 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:29.803390    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"]
Oct 30 22:01:29.803829 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:29.803702    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" containerID="cri-o://d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb" gracePeriod=30
Oct 30 22:01:32.233212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.233106    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"]
Oct 30 22:01:32.234054 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.234023    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh" podUID="2d0b762c-db09-4e87-9021-5e34a42eb86e" containerName="maas-api" containerID="cri-o://e2296cd644fd51e18bf62b3233ca586139b0413257bb5886d49b20ce3109dc02" gracePeriod=30
Oct 30 22:01:32.545401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.545371    4282 generic.go:356] "Generic (PLEG): container finished" podID="2d0b762c-db09-4e87-9021-5e34a42eb86e" containerID="e2296cd644fd51e18bf62b3233ca586139b0413257bb5886d49b20ce3109dc02" exitCode=0
Oct 30 22:01:32.545587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.545447    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh" event={"ID":"2d0b762c-db09-4e87-9021-5e34a42eb86e","Type":"ContainerDied","Data":"e2296cd644fd51e18bf62b3233ca586139b0413257bb5886d49b20ce3109dc02"}
Oct 30 22:01:32.770490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.758162    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 22:01:32.929999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.929964    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0b762c-db09-4e87-9021-5e34a42eb86e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0b762c-db09-4e87-9021-5e34a42eb86e-openshift-service-ca\") pod \"2d0b762c-db09-4e87-9021-5e34a42eb86e\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") "
Oct 30 22:01:32.930265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.930245    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls\") pod \"2d0b762c-db09-4e87-9021-5e34a42eb86e\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") "
Oct 30 22:01:32.930370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.930329    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0b762c-db09-4e87-9021-5e34a42eb86e-kube-api-access-8szrz\" (UniqueName: \"kubernetes.io/projected/2d0b762c-db09-4e87-9021-5e34a42eb86e-kube-api-access-8szrz\") pod \"2d0b762c-db09-4e87-9021-5e34a42eb86e\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") "
Oct 30 22:01:32.930443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.930412    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-metrics-certs\") pod \"2d0b762c-db09-4e87-9021-5e34a42eb86e\" (UID: \"2d0b762c-db09-4e87-9021-5e34a42eb86e\") "
Oct 30 22:01:32.930540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.930326    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0b762c-db09-4e87-9021-5e34a42eb86e-openshift-service-ca" pod "2d0b762c-db09-4e87-9021-5e34a42eb86e" (UID: "2d0b762c-db09-4e87-9021-5e34a42eb86e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:32.931102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.931078    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0b762c-db09-4e87-9021-5e34a42eb86e-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:32.933921 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.933882    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-metrics-certs" pod "2d0b762c-db09-4e87-9021-5e34a42eb86e" (UID: "2d0b762c-db09-4e87-9021-5e34a42eb86e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:32.934033 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.934009    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls" pod "2d0b762c-db09-4e87-9021-5e34a42eb86e" (UID: "2d0b762c-db09-4e87-9021-5e34a42eb86e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:32.934230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:32.934205    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0b762c-db09-4e87-9021-5e34a42eb86e-kube-api-access-8szrz" pod "2d0b762c-db09-4e87-9021-5e34a42eb86e" (UID: "2d0b762c-db09-4e87-9021-5e34a42eb86e"). InnerVolumeSpecName "kube-api-access-8szrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:33.057244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:33.057209    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:33.059727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:33.059695    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8szrz\" (UniqueName: \"kubernetes.io/projected/2d0b762c-db09-4e87-9021-5e34a42eb86e-kube-api-access-8szrz\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:33.059727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:33.059730    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d0b762c-db09-4e87-9021-5e34a42eb86e-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:33.553146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:33.553117    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"
Oct 30 22:01:33.553605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:33.553116    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh" event={"ID":"2d0b762c-db09-4e87-9021-5e34a42eb86e","Type":"ContainerDied","Data":"2d0298c97d8e12a3499a992a5136e8ae36403009ab42d5ba79e2e256cdf00d81"}
Oct 30 22:01:33.553605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:33.553251    4282 scope.go:122] "RemoveContainer" containerID="e2296cd644fd51e18bf62b3233ca586139b0413257bb5886d49b20ce3109dc02"
Oct 30 22:01:33.603968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:33.603920    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"]
Oct 30 22:01:33.607812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:33.607791    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3654d5-bd6d548db-fnnkh"]
Oct 30 22:01:34.204821 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:34.204789    4282 patch_prober.go:29] interesting pod/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:01:34.205016 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:34.204846    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:01:34.423578 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:34.423541    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"]
Oct 30 22:01:34.651217 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:34.651129    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:01:35.250976 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.250948    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 22:01:35.411328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411238    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-data\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.411328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411320    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-envoy\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.411599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411378    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-podinfo\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.411599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411407    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-kube-api-access-prwf5\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-kube-api-access-prwf5\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.411599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411442    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-credential-socket\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.411756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411643    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-token\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-token\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.411756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411699    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-certs\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.411849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411837    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istiod-ca-cert\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.411897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.411877    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-data" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:35.412298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.412268    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istiod-ca-cert" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:35.412586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.412563    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-certs" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:35.412784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.412765    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-socket\") pod \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\" (UID: \"0c43be2f-8c29-48e2-b96b-0dec00fa03ff\") "
Oct 30 22:01:35.413151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.412772    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-credential-socket" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:35.413151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.412982    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-socket" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:35.413817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.413793    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.413943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.413930    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.414032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.414020    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.414115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.414105    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.414190 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.414179    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.414645 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.414622    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-podinfo" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:01:35.414754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.414740    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-envoy" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:35.414902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.414880    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-kube-api-access-prwf5" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "kube-api-access-prwf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:35.416287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.416266    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-token" pod "0c43be2f-8c29-48e2-b96b-0dec00fa03ff" (UID: "0c43be2f-8c29-48e2-b96b-0dec00fa03ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:35.500000 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.499960    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0b762c-db09-4e87-9021-5e34a42eb86e" path="/var/lib/kubelet/pods/2d0b762c-db09-4e87-9021-5e34a42eb86e/volumes"
Oct 30 22:01:35.515413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.515380    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.515584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.515414    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.515584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.515428    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prwf5\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-kube-api-access-prwf5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.515584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.515441    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c43be2f-8c29-48e2-b96b-0dec00fa03ff-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:35.587498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.586706    4282 generic.go:356] "Generic (PLEG): container finished" podID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerID="d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb" exitCode=0
Oct 30 22:01:35.587498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.586853    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" event={"ID":"0c43be2f-8c29-48e2-b96b-0dec00fa03ff","Type":"ContainerDied","Data":"d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb"}
Oct 30 22:01:35.587498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.586904    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w" event={"ID":"0c43be2f-8c29-48e2-b96b-0dec00fa03ff","Type":"ContainerDied","Data":"ef5793d2d2e30e4bfb68080e28a3c25aa1ee3441ada8bfc1850da1f1230060ee"}
Oct 30 22:01:35.587498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.586933    4282 scope.go:122] "RemoveContainer" containerID="d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb"
Oct 30 22:01:35.587498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.587112    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"
Oct 30 22:01:35.604280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.604248    4282 scope.go:122] "RemoveContainer" containerID="8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795"
Oct 30 22:01:35.618498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.618299    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"]
Oct 30 22:01:35.621202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.621172    4282 scope.go:122] "RemoveContainer" containerID="d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb"
Oct 30 22:01:35.621700 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:35.621669    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb\": container with ID starting with d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb not found: ID does not exist" containerID="d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb"
Oct 30 22:01:35.621796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.621712    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb"} err="failed to get container status \"d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb\": rpc error: code = NotFound desc = could not find container \"d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb\": container with ID starting with d939e74a287a81683e86f02b03730eab32492c1baba8297053a59964bdf583bb not found: ID does not exist"
Oct 30 22:01:35.621796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.621739    4282 scope.go:122] "RemoveContainer" containerID="8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795"
Oct 30 22:01:35.622186 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:35.622080    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795\": container with ID starting with 8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795 not found: ID does not exist" containerID="8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795"
Oct 30 22:01:35.622186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.622111    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795"} err="failed to get container status \"8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795\": rpc error: code = NotFound desc = could not find container \"8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795\": container with ID starting with 8490496fdf2de9592e82fd35eada542b3fbdc455ea4d935427d43f41278f1795 not found: ID does not exist"
Oct 30 22:01:35.623850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:35.623834    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c614f9-openshift-default-84bfd798bf-mpd2w"]
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.329956    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"]
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.330962    4282 cpu_manager.go:414] "containerMap: removing container" podUID="2d0b762c-db09-4e87-9021-5e34a42eb86e" containerName="maas-api"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.330984    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0b762c-db09-4e87-9021-5e34a42eb86e" containerName="maas-api"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331016    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331023    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331044    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331053    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331085    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331092    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331268    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0b762c-db09-4e87-9021-5e34a42eb86e" containerName="maas-api"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331285    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331299    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331310    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy"
Oct 30 22:01:36.340525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.331325    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" containerName="istio-proxy"
Oct 30 22:01:36.594545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.593627    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.627500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.625638    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"]
Oct 30 22:01:36.679494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.679385    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c097ae-serving-cert\""
Oct 30 22:01:36.838324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.837885    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63444212-7be1-4758-b943-9d5be34f4d1e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.838324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.838013    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvlzf\" (UniqueName: \"kubernetes.io/projected/63444212-7be1-4758-b943-9d5be34f4d1e-kube-api-access-jvlzf\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.838324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.838056    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-metrics-certs\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.838324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.838208    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.939540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.939507    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63444212-7be1-4758-b943-9d5be34f4d1e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.939723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.939576    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvlzf\" (UniqueName: \"kubernetes.io/projected/63444212-7be1-4758-b943-9d5be34f4d1e-kube-api-access-jvlzf\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.939723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.939600    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-metrics-certs\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.939723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.939670    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.940311 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.940286    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63444212-7be1-4758-b943-9d5be34f4d1e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.942225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.942202    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-metrics-certs\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.942331 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.942248    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:36.978490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:36.978231    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvlzf\" (UniqueName: \"kubernetes.io/projected/63444212-7be1-4758-b943-9d5be34f4d1e-kube-api-access-jvlzf\") pod \"maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:37.107989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:37.107952    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:37.646858 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:01:37.646793    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63444212_7be1_4758_b943_9d5be34f4d1e.slice/crio-dabcbf5828ed485a8ddeecdfc6814a50da4f3023ec48b886cbee102db6c2af0b WatchSource:0}: Error finding container dabcbf5828ed485a8ddeecdfc6814a50da4f3023ec48b886cbee102db6c2af0b: Status 404 returned error can't find the container with id dabcbf5828ed485a8ddeecdfc6814a50da4f3023ec48b886cbee102db6c2af0b
Oct 30 22:01:37.648272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:37.648244    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c43be2f-8c29-48e2-b96b-0dec00fa03ff" path="/var/lib/kubelet/pods/0c43be2f-8c29-48e2-b96b-0dec00fa03ff/volumes"
Oct 30 22:01:37.650941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:37.650921    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"]
Oct 30 22:01:37.810831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:37.810739    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4" event={"ID":"63444212-7be1-4758-b943-9d5be34f4d1e","Type":"ContainerStarted","Data":"dabcbf5828ed485a8ddeecdfc6814a50da4f3023ec48b886cbee102db6c2af0b"}
Oct 30 22:01:38.748213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:38.748146    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"]
Oct 30 22:01:38.821904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:38.821861    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4" event={"ID":"63444212-7be1-4758-b943-9d5be34f4d1e","Type":"ContainerStarted","Data":"ec90c3f39f1df786b0a9d6ca7d86f25c59772a30ccce4f7451ea0beb32963398"}
Oct 30 22:01:38.822092 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:38.821956    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:38.845650 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:38.845592    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4" podStartSLOduration=2.289214736 podStartE2EDuration="2.845571006s" totalImagesPullingTime="556.35627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:38.84205152 +0000 UTC m=+1317.500147754" watchObservedRunningTime="2026-10-30 22:01:38.845571006 +0000 UTC m=+1317.503667234"
Oct 30 22:01:39.241312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.239815    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": read tcp 10.133.0.2:32954->10.133.0.124:15021: read: connection reset by peer" start-of-body=
Oct 30 22:01:39.241312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.239864    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": read tcp 10.133.0.2:32954->10.133.0.124:15021: read: connection reset by peer"
Oct 30 22:01:39.429567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.424423    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4" podUID="200c1579-38d3-4168-8cb1-81de4feb4793" containerName="payload-pre-processing" containerID="cri-o://d199360064462638e773d75ec3d97a832a21aa7c38f9103c7a2a643dd85f45ed" gracePeriod=25
Oct 30 22:01:39.863550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.855560    4282 generic.go:356] "Generic (PLEG): container finished" podID="200c1579-38d3-4168-8cb1-81de4feb4793" containerID="d199360064462638e773d75ec3d97a832a21aa7c38f9103c7a2a643dd85f45ed" exitCode=0
Oct 30 22:01:39.880533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.855938    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4" event={"ID":"200c1579-38d3-4168-8cb1-81de4feb4793","Type":"ContainerDied","Data":"d199360064462638e773d75ec3d97a832a21aa7c38f9103c7a2a643dd85f45ed"}
Oct 30 22:01:39.880533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.877380    4282 generic.go:356] "Generic (PLEG): container finished" podID="16347946-2949-44e3-84fd-3621c321ad08" containerID="123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d" exitCode=0
Oct 30 22:01:39.880533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.878830    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" event={"ID":"16347946-2949-44e3-84fd-3621c321ad08","Type":"ContainerDied","Data":"123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d"}
Oct 30 22:01:39.880533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.879370    4282 scope.go:122] "RemoveContainer" containerID="123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d"
Oct 30 22:01:39.882542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.882501    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:39.882712 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.882671    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:39.882814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.882782    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:39.983648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:39.983618    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:01:40.038560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.038434    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/200c1579-38d3-4168-8cb1-81de4feb4793-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200c1579-38d3-4168-8cb1-81de4feb4793-plugins-config-volume\") pod \"200c1579-38d3-4168-8cb1-81de4feb4793\" (UID: \"200c1579-38d3-4168-8cb1-81de4feb4793\") "
Oct 30 22:01:40.038776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.038681    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/200c1579-38d3-4168-8cb1-81de4feb4793-kube-api-access-v48js\" (UniqueName: \"kubernetes.io/projected/200c1579-38d3-4168-8cb1-81de4feb4793-kube-api-access-v48js\") pod \"200c1579-38d3-4168-8cb1-81de4feb4793\" (UID: \"200c1579-38d3-4168-8cb1-81de4feb4793\") "
Oct 30 22:01:40.039055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.039022    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/200c1579-38d3-4168-8cb1-81de4feb4793-plugins-config-volume" pod "200c1579-38d3-4168-8cb1-81de4feb4793" (UID: "200c1579-38d3-4168-8cb1-81de4feb4793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:40.040423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.040391    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/200c1579-38d3-4168-8cb1-81de4feb4793-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:40.042872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.042849    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/200c1579-38d3-4168-8cb1-81de4feb4793-kube-api-access-v48js" pod "200c1579-38d3-4168-8cb1-81de4feb4793" (UID: "200c1579-38d3-4168-8cb1-81de4feb4793"). InnerVolumeSpecName "kube-api-access-v48js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:40.148543 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.144420    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v48js\" (UniqueName: \"kubernetes.io/projected/200c1579-38d3-4168-8cb1-81de4feb4793-kube-api-access-v48js\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:40.907491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.907389    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4" event={"ID":"200c1579-38d3-4168-8cb1-81de4feb4793","Type":"ContainerDied","Data":"6b6c5e7ad012ca7a0d0190eaa62cb1ce0d5adae70407b413c6fe458a0868e72a"}
Oct 30 22:01:40.907491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.907447    4282 scope.go:122] "RemoveContainer" containerID="d199360064462638e773d75ec3d97a832a21aa7c38f9103c7a2a643dd85f45ed"
Oct 30 22:01:40.907954 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.907503    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"
Oct 30 22:01:40.910134 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.910113    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" event={"ID":"16347946-2949-44e3-84fd-3621c321ad08","Type":"ContainerStarted","Data":"89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23"}
Oct 30 22:01:40.948752 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:40.948724    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"]
Oct 30 22:01:41.001497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:41.000509    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ecffad-8c688575-hnpw4"]
Oct 30 22:01:41.489817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:41.489784    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="200c1579-38d3-4168-8cb1-81de4feb4793" path="/var/lib/kubelet/pods/200c1579-38d3-4168-8cb1-81de4feb4793/volumes"
Oct 30 22:01:41.508915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:41.508863    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:01:41.509057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:41.509037    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:01:41.509127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:41.509084    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:01:42.509381 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:42.509343    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:01:42.509870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:42.509412    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:01:43.108811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.108773    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"]
Oct 30 22:01:43.109096 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.109045    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv" podUID="72193abe-3cdb-4303-a216-6c3a64e5925b" containerName="maas-api" containerID="cri-o://f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa" gracePeriod=30
Oct 30 22:01:43.190792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.190757    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv" podUID="72193abe-3cdb-4303-a216-6c3a64e5925b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.121:8443/health\": dial tcp 10.133.0.121:8443: connect: connection refused"
Oct 30 22:01:43.676606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.676567    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:01:43.677121 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.676643    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:01:43.892807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.892781    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:01:43.907123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.907099    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-metrics-certs\") pod \"72193abe-3cdb-4303-a216-6c3a64e5925b\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") "
Oct 30 22:01:43.907235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.907219    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72193abe-3cdb-4303-a216-6c3a64e5925b-kube-api-access-ftcfc\" (UniqueName: \"kubernetes.io/projected/72193abe-3cdb-4303-a216-6c3a64e5925b-kube-api-access-ftcfc\") pod \"72193abe-3cdb-4303-a216-6c3a64e5925b\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") "
Oct 30 22:01:43.907576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.907548    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72193abe-3cdb-4303-a216-6c3a64e5925b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72193abe-3cdb-4303-a216-6c3a64e5925b-openshift-service-ca\") pod \"72193abe-3cdb-4303-a216-6c3a64e5925b\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") "
Oct 30 22:01:43.907662 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.907587    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls\") pod \"72193abe-3cdb-4303-a216-6c3a64e5925b\" (UID: \"72193abe-3cdb-4303-a216-6c3a64e5925b\") "
Oct 30 22:01:43.907998 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.907948    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72193abe-3cdb-4303-a216-6c3a64e5925b-openshift-service-ca" pod "72193abe-3cdb-4303-a216-6c3a64e5925b" (UID: "72193abe-3cdb-4303-a216-6c3a64e5925b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:43.908118 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.908100    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72193abe-3cdb-4303-a216-6c3a64e5925b-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:43.909605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.909583    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-metrics-certs" pod "72193abe-3cdb-4303-a216-6c3a64e5925b" (UID: "72193abe-3cdb-4303-a216-6c3a64e5925b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:43.909739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.909719    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72193abe-3cdb-4303-a216-6c3a64e5925b-kube-api-access-ftcfc" pod "72193abe-3cdb-4303-a216-6c3a64e5925b" (UID: "72193abe-3cdb-4303-a216-6c3a64e5925b"). InnerVolumeSpecName "kube-api-access-ftcfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:43.910141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:43.910116    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls" pod "72193abe-3cdb-4303-a216-6c3a64e5925b" (UID: "72193abe-3cdb-4303-a216-6c3a64e5925b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:44.006528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.006377    4282 generic.go:356] "Generic (PLEG): container finished" podID="72193abe-3cdb-4303-a216-6c3a64e5925b" containerID="f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa" exitCode=0
Oct 30 22:01:44.006528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.006446    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"
Oct 30 22:01:44.006528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.006462    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv" event={"ID":"72193abe-3cdb-4303-a216-6c3a64e5925b","Type":"ContainerDied","Data":"f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa"}
Oct 30 22:01:44.006838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.006541    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv" event={"ID":"72193abe-3cdb-4303-a216-6c3a64e5925b","Type":"ContainerDied","Data":"9be950c8b82d4874f21f6030b4e0814b4dd3ed542c3abc190cd8c8c552426456"}
Oct 30 22:01:44.006838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.006577    4282 scope.go:122] "RemoveContainer" containerID="f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa"
Oct 30 22:01:44.009678 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.009651    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftcfc\" (UniqueName: \"kubernetes.io/projected/72193abe-3cdb-4303-a216-6c3a64e5925b-kube-api-access-ftcfc\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:44.009825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.009683    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:44.009825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.009698    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72193abe-3cdb-4303-a216-6c3a64e5925b-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:44.019641 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.019615    4282 scope.go:122] "RemoveContainer" containerID="f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa"
Oct 30 22:01:44.020021 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:44.019980    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa\": container with ID starting with f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa not found: ID does not exist" containerID="f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa"
Oct 30 22:01:44.020101 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.020023    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa"} err="failed to get container status \"f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa\": rpc error: code = NotFound desc = could not find container \"f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa\": container with ID starting with f637f41a46f2961f47e6e85b28a661df4807b3e40483790737dee4d5bf8ac0fa not found: ID does not exist"
Oct 30 22:01:44.040137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.040103    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"]
Oct 30 22:01:44.086346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.086312    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ecffad-84465987f6-bqssv"]
Oct 30 22:01:44.550823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.547675    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:01:44.550823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.547749    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:01:44.886301 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:44.886219    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:01:45.146413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.146343    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"]
Oct 30 22:01:45.147023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.147008    4282 cpu_manager.go:414] "containerMap: removing container" podUID="72193abe-3cdb-4303-a216-6c3a64e5925b" containerName="maas-api"
Oct 30 22:01:45.147080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.147024    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72193abe-3cdb-4303-a216-6c3a64e5925b" containerName="maas-api"
Oct 30 22:01:45.147080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.147042    4282 cpu_manager.go:414] "containerMap: removing container" podUID="200c1579-38d3-4168-8cb1-81de4feb4793" containerName="payload-pre-processing"
Oct 30 22:01:45.147080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.147047    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="200c1579-38d3-4168-8cb1-81de4feb4793" containerName="payload-pre-processing"
Oct 30 22:01:45.147080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.147072    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy"
Oct 30 22:01:45.147080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.147076    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7646623d-2c6a-4b60-90f8-31b67cb69c05" containerName="istio-proxy"
Oct 30 22:01:45.147231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.147166    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="72193abe-3cdb-4303-a216-6c3a64e5925b" containerName="maas-api"
Oct 30 22:01:45.147231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.147176    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="200c1579-38d3-4168-8cb1-81de4feb4793" containerName="payload-pre-processing"
Oct 30 22:01:45.170610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.152308    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.170610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.167756    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a1ead7-serving-cert\""
Oct 30 22:01:45.192957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.192932    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"]
Oct 30 22:01:45.248929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.248903    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.249453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.249030    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.249453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.249135    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6z5r\" (UniqueName: \"kubernetes.io/projected/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-kube-api-access-d6z5r\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.249453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.249299    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-metrics-certs\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.361734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.356587    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6z5r\" (UniqueName: \"kubernetes.io/projected/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-kube-api-access-d6z5r\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.361734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.356852    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-metrics-certs\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.361734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.357001    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.361734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.357178    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.361734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.358251    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.370489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.367156    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-metrics-certs\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.415951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.412089    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.415951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.414752    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6z5r\" (UniqueName: \"kubernetes.io/projected/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-kube-api-access-d6z5r\") pod \"maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.512623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.511201    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 22:01:45.512623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.511269    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 22:01:45.512623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.511836    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:45.520038 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.519966    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72193abe-3cdb-4303-a216-6c3a64e5925b" path="/var/lib/kubelet/pods/72193abe-3cdb-4303-a216-6c3a64e5925b/volumes"
Oct 30 22:01:45.912494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:45.900630    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"]
Oct 30 22:01:46.036338 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:46.036306    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj" event={"ID":"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc","Type":"ContainerStarted","Data":"7d148386458ef62a42945daa8b8e9018f7a95996d652dc2f8e0c63f8d0e1eeaf"}
Oct 30 22:01:46.585449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:46.585280    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:01:46.587437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:46.586197    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:01:46.589031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:46.589010    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:01:47.045095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.045060    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj" event={"ID":"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc","Type":"ContainerStarted","Data":"3c34094ffea8a2e8a9e4edcedddd41e8c8f73a7dad4e3cf36f4a620ce0f8897a"}
Oct 30 22:01:47.045649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.045205    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:47.070138 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.070087    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj" podStartSLOduration=1.6210662409999999 podStartE2EDuration="2.070071807s" totalImagesPullingTime="449.005566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:47.068436243 +0000 UTC m=+1325.726532467" watchObservedRunningTime="2026-10-30 22:01:47.070071807 +0000 UTC m=+1325.728168029"
Oct 30 22:01:47.855949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.855918    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"]
Oct 30 22:01:47.856196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.856169    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd" podUID="c780ada2-2ab1-4eb3-bd75-939a40a1eef7" containerName="maas-api" containerID="cri-o://772e55efbac1218802d40f412ab110780695841512664fdea7083c840e4dd7b0" gracePeriod=30
Oct 30 22:01:47.990230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.990200    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"]
Oct 30 22:01:47.990459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.990433    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc" podUID="520bdd2e-92e3-4570-aecd-2932ec9b21f4" containerName="payload-processing" containerID="cri-o://fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0" gracePeriod=30
Oct 30 22:01:47.998775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.998743    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"]
Oct 30 22:01:47.999008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:47.998975    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr" podUID="c95cc705-3ffa-4604-97d5-84f50dc54d2c" containerName="payload-pre-processing" containerID="cri-o://6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1" gracePeriod=30
Oct 30 22:01:48.054354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.054327    4282 generic.go:356] "Generic (PLEG): container finished" podID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerID="9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002" exitCode=0
Oct 30 22:01:48.054795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.054396    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" event={"ID":"519c0cc2-aeef-4a66-b158-f6e703e5d0da","Type":"ContainerDied","Data":"9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002"}
Oct 30 22:01:48.054909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.054879    4282 scope.go:122] "RemoveContainer" containerID="9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002"
Oct 30 22:01:48.056385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.056363    4282 generic.go:356] "Generic (PLEG): container finished" podID="c780ada2-2ab1-4eb3-bd75-939a40a1eef7" containerID="772e55efbac1218802d40f412ab110780695841512664fdea7083c840e4dd7b0" exitCode=0
Oct 30 22:01:48.056510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.056448    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd" event={"ID":"c780ada2-2ab1-4eb3-bd75-939a40a1eef7","Type":"ContainerDied","Data":"772e55efbac1218802d40f412ab110780695841512664fdea7083c840e4dd7b0"}
Oct 30 22:01:48.056660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.056633    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:48.056718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.056703    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:48.056769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.056753    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:48.188320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.188290    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:01:48.283533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.266405    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-openshift-service-ca\") pod \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") "
Oct 30 22:01:48.283533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.266746    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-metrics-certs\") pod \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") "
Oct 30 22:01:48.283533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.266908    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-maas-api-tls\") pod \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") "
Oct 30 22:01:48.283533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.266942    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-kube-api-access-6kwk9\" (UniqueName: \"kubernetes.io/projected/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-kube-api-access-6kwk9\") pod \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\" (UID: \"c780ada2-2ab1-4eb3-bd75-939a40a1eef7\") "
Oct 30 22:01:48.283533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.277789    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-openshift-service-ca" pod "c780ada2-2ab1-4eb3-bd75-939a40a1eef7" (UID: "c780ada2-2ab1-4eb3-bd75-939a40a1eef7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:48.382794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.374012    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-maas-api-tls" pod "c780ada2-2ab1-4eb3-bd75-939a40a1eef7" (UID: "c780ada2-2ab1-4eb3-bd75-939a40a1eef7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:48.382794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.375440    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:48.382794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.375534    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:48.526743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.513751    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-metrics-certs" pod "c780ada2-2ab1-4eb3-bd75-939a40a1eef7" (UID: "c780ada2-2ab1-4eb3-bd75-939a40a1eef7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:48.538431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.529457    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-kube-api-access-6kwk9" pod "c780ada2-2ab1-4eb3-bd75-939a40a1eef7" (UID: "c780ada2-2ab1-4eb3-bd75-939a40a1eef7"). InnerVolumeSpecName "kube-api-access-6kwk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:48.588325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.585916    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:48.588638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.588605    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwk9\" (UniqueName: \"kubernetes.io/projected/c780ada2-2ab1-4eb3-bd75-939a40a1eef7-kube-api-access-6kwk9\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:48.610890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.610451    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:01:48.708393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.708279    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520bdd2e-92e3-4570-aecd-2932ec9b21f4-kube-api-access-jj8v9\" (UniqueName: \"kubernetes.io/projected/520bdd2e-92e3-4570-aecd-2932ec9b21f4-kube-api-access-jj8v9\") pod \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\" (UID: \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\") "
Oct 30 22:01:48.714806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.714758    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520bdd2e-92e3-4570-aecd-2932ec9b21f4-plugins-config-volume" pod "520bdd2e-92e3-4570-aecd-2932ec9b21f4" (UID: "520bdd2e-92e3-4570-aecd-2932ec9b21f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:48.718445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.717546    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520bdd2e-92e3-4570-aecd-2932ec9b21f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520bdd2e-92e3-4570-aecd-2932ec9b21f4-plugins-config-volume\") pod \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\" (UID: \"520bdd2e-92e3-4570-aecd-2932ec9b21f4\") "
Oct 30 22:01:48.718445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.718308    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/520bdd2e-92e3-4570-aecd-2932ec9b21f4-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:48.723491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.723393    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:01:48.771110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.771058    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520bdd2e-92e3-4570-aecd-2932ec9b21f4-kube-api-access-jj8v9" pod "520bdd2e-92e3-4570-aecd-2932ec9b21f4" (UID: "520bdd2e-92e3-4570-aecd-2932ec9b21f4"). InnerVolumeSpecName "kube-api-access-jj8v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:48.819981 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.819939    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95cc705-3ffa-4604-97d5-84f50dc54d2c-kube-api-access-hpq2q\" (UniqueName: \"kubernetes.io/projected/c95cc705-3ffa-4604-97d5-84f50dc54d2c-kube-api-access-hpq2q\") pod \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\" (UID: \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\") "
Oct 30 22:01:48.820144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.820074    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95cc705-3ffa-4604-97d5-84f50dc54d2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95cc705-3ffa-4604-97d5-84f50dc54d2c-plugins-config-volume\") pod \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\" (UID: \"c95cc705-3ffa-4604-97d5-84f50dc54d2c\") "
Oct 30 22:01:48.820605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.820576    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95cc705-3ffa-4604-97d5-84f50dc54d2c-plugins-config-volume" pod "c95cc705-3ffa-4604-97d5-84f50dc54d2c" (UID: "c95cc705-3ffa-4604-97d5-84f50dc54d2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:48.820964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.820923    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj8v9\" (UniqueName: \"kubernetes.io/projected/520bdd2e-92e3-4570-aecd-2932ec9b21f4-kube-api-access-jj8v9\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:48.820964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.820954    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95cc705-3ffa-4604-97d5-84f50dc54d2c-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:48.822448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.822419    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95cc705-3ffa-4604-97d5-84f50dc54d2c-kube-api-access-hpq2q" pod "c95cc705-3ffa-4604-97d5-84f50dc54d2c" (UID: "c95cc705-3ffa-4604-97d5-84f50dc54d2c"). InnerVolumeSpecName "kube-api-access-hpq2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:48.922732 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:48.922700    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpq2q\" (UniqueName: \"kubernetes.io/projected/c95cc705-3ffa-4604-97d5-84f50dc54d2c-kube-api-access-hpq2q\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:49.065730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.065633    4282 generic.go:356] "Generic (PLEG): container finished" podID="520bdd2e-92e3-4570-aecd-2932ec9b21f4" containerID="fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0" exitCode=0
Oct 30 22:01:49.065730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.065684    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"
Oct 30 22:01:49.065730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.065717    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc" event={"ID":"520bdd2e-92e3-4570-aecd-2932ec9b21f4","Type":"ContainerDied","Data":"fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0"}
Oct 30 22:01:49.066328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.065776    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc" event={"ID":"520bdd2e-92e3-4570-aecd-2932ec9b21f4","Type":"ContainerDied","Data":"b9ce50e57fd672437715230942256b1fc263a657177e521866ecc053badeb791"}
Oct 30 22:01:49.066328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.065806    4282 scope.go:122] "RemoveContainer" containerID="fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0"
Oct 30 22:01:49.071445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.071415    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" event={"ID":"519c0cc2-aeef-4a66-b158-f6e703e5d0da","Type":"ContainerStarted","Data":"95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183"}
Oct 30 22:01:49.072992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.072966    4282 generic.go:356] "Generic (PLEG): container finished" podID="c95cc705-3ffa-4604-97d5-84f50dc54d2c" containerID="6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1" exitCode=0
Oct 30 22:01:49.073088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.073010    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"
Oct 30 22:01:49.073088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.073046    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr" event={"ID":"c95cc705-3ffa-4604-97d5-84f50dc54d2c","Type":"ContainerDied","Data":"6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1"}
Oct 30 22:01:49.073170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.073111    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr" event={"ID":"c95cc705-3ffa-4604-97d5-84f50dc54d2c","Type":"ContainerDied","Data":"f4773ed8f05f6fb6781bcad90d58cf04cc7cc66f50b2213255fc7a8c80273e03"}
Oct 30 22:01:49.074833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.074796    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd" event={"ID":"c780ada2-2ab1-4eb3-bd75-939a40a1eef7","Type":"ContainerDied","Data":"b4d81d58539e0e9d0b2d3182b5ef43078954e74ef0847d5096a67c98042595c7"}
Oct 30 22:01:49.074938 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.074889    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"
Oct 30 22:01:49.077716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.077694    4282 scope.go:122] "RemoveContainer" containerID="fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0"
Oct 30 22:01:49.078030 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:49.078006    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0\": container with ID starting with fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0 not found: ID does not exist" containerID="fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0"
Oct 30 22:01:49.078109 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.078035    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0"} err="failed to get container status \"fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0\": rpc error: code = NotFound desc = could not find container \"fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0\": container with ID starting with fd3168660c21c13e2232551c0d579cefd1930d8ea32edfa0580a5609e84fc9d0 not found: ID does not exist"
Oct 30 22:01:49.078109 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.078052    4282 scope.go:122] "RemoveContainer" containerID="6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1"
Oct 30 22:01:49.087869 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.087852    4282 scope.go:122] "RemoveContainer" containerID="6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1"
Oct 30 22:01:49.088141 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:49.088123    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1\": container with ID starting with 6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1 not found: ID does not exist" containerID="6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1"
Oct 30 22:01:49.088199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.088149    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1"} err="failed to get container status \"6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1\": rpc error: code = NotFound desc = could not find container \"6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1\": container with ID starting with 6c41c8a8e4ec638df77cdd2030fe0f263a7a1ea09bb7a4c58767ad9b957f1df1 not found: ID does not exist"
Oct 30 22:01:49.088199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.088165    4282 scope.go:122] "RemoveContainer" containerID="772e55efbac1218802d40f412ab110780695841512664fdea7083c840e4dd7b0"
Oct 30 22:01:49.118357 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.118335    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"]
Oct 30 22:01:49.127127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.127103    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5289f-56c674489f-fq2qr"]
Oct 30 22:01:49.150308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.150287    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"]
Oct 30 22:01:49.158061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.158033    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e5289f-5b4dcc9dd5-mx2dd"]
Oct 30 22:01:49.176326 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.176304    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"]
Oct 30 22:01:49.188569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.188549    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5289f-8565f45d74-rlnrc"]
Oct 30 22:01:49.442724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.442014    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:01:49.500513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.494982    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520bdd2e-92e3-4570-aecd-2932ec9b21f4" path="/var/lib/kubelet/pods/520bdd2e-92e3-4570-aecd-2932ec9b21f4/volumes"
Oct 30 22:01:49.500513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.495683    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c780ada2-2ab1-4eb3-bd75-939a40a1eef7" path="/var/lib/kubelet/pods/c780ada2-2ab1-4eb3-bd75-939a40a1eef7/volumes"
Oct 30 22:01:49.500513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.496233    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95cc705-3ffa-4604-97d5-84f50dc54d2c" path="/var/lib/kubelet/pods/c95cc705-3ffa-4604-97d5-84f50dc54d2c/volumes"
Oct 30 22:01:49.619569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.614933    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:01:49.619569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.615057    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:01:49.619569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:49.615097    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:01:50.613497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:50.612839    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:01:50.613497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:50.612903    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:01:51.613160 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:51.613121    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:01:51.613356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:51.613170    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:01:52.613015 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:52.612980    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:01:52.613574 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:52.613041    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:01:53.088665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.088634    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.381071    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"]
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.382625    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c780ada2-2ab1-4eb3-bd75-939a40a1eef7" containerName="maas-api"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.382648    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c780ada2-2ab1-4eb3-bd75-939a40a1eef7" containerName="maas-api"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.382673    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c95cc705-3ffa-4604-97d5-84f50dc54d2c" containerName="payload-pre-processing"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.382682    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95cc705-3ffa-4604-97d5-84f50dc54d2c" containerName="payload-pre-processing"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.382769    4282 cpu_manager.go:414] "containerMap: removing container" podUID="520bdd2e-92e3-4570-aecd-2932ec9b21f4" containerName="payload-processing"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.382777    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520bdd2e-92e3-4570-aecd-2932ec9b21f4" containerName="payload-processing"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.383094    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="520bdd2e-92e3-4570-aecd-2932ec9b21f4" containerName="payload-processing"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.383126    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c780ada2-2ab1-4eb3-bd75-939a40a1eef7" containerName="maas-api"
Oct 30 22:01:53.385553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.383146    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95cc705-3ffa-4604-97d5-84f50dc54d2c" containerName="payload-pre-processing"
Oct 30 22:01:53.413197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.408740    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:01:53.413197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.409447    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.430422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.420256    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bca8c9-openshift-default-dockercfg-9496l\""
Oct 30 22:01:53.430422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.425856    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"]
Oct 30 22:01:53.508846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.497939    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-credential-socket\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.509174    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"]
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.520586    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eda7d94-5787-4a6d-97ce-bd2337998a88-istiod-ca-cert\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.520681    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-certs\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.520742    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-token\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.520912    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-podinfo\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.520953    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-socket\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.521021    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-data\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.521087    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4n8\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-kube-api-access-mk4n8\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.524509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.521128    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-envoy\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.650555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.634951    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 22:01:53.693502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.667762    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 22:01:53.701498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.694505    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-credential-socket\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.701794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.701762    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eda7d94-5787-4a6d-97ce-bd2337998a88-istiod-ca-cert\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.742493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.701961    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-certs\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.780491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.768149    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-token\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.780491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.768307    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-podinfo\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.780491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.768347    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-socket\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.780491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.768398    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-data\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.780491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.768486    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4n8\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-kube-api-access-mk4n8\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.780491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.768575    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-envoy\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.831526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.721768    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-credential-socket\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.831526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.722378    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eda7d94-5787-4a6d-97ce-bd2337998a88-istiod-ca-cert\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.833499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.831985    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-podinfo\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.833499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.832291    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-socket\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.833499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.768099    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-certs\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.833499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.833116    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-data\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.878491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.844605    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx" podUID="7929d5a4-5bdb-4fac-82a0-514d24b190c6" containerName="main" containerID="cri-o://01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975" gracePeriod=45
Oct 30 22:01:53.943527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.936415    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx" podUID="7929d5a4-5bdb-4fac-82a0-514d24b190c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.113:8000/ready\": dial tcp 10.133.0.113:8000: connect: connection refused"
Oct 30 22:01:53.943527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.939653    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-token\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.943527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.940065    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-envoy\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:53.967446 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:53.967417    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4n8\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-kube-api-access-mk4n8\") pod \"e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") " pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:54.067516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.063816    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:54.136998 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.136946    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy" containerID="cri-o://95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183" gracePeriod=30
Oct 30 22:01:54.516929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.516896    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"]
Oct 30 22:01:54.520700 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:01:54.520279    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eda7d94_5787_4a6d_97ce_bd2337998a88.slice/crio-e537eda573b6d23c51d7fa221f96ed91975970be243d80c26145a5f59f29d5f8 WatchSource:0}: Error finding container e537eda573b6d23c51d7fa221f96ed91975970be243d80c26145a5f59f29d5f8: Status 404 returned error can't find the container with id e537eda573b6d23c51d7fa221f96ed91975970be243d80c26145a5f59f29d5f8
Oct 30 22:01:54.527717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.527674    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:54.527826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.527776    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:54.527912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.527822    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:01:54.664964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.664933    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 22:01:54.869260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.849175    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-home\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-home\") pod \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") "
Oct 30 22:01:54.869260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.849327    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7929d5a4-5bdb-4fac-82a0-514d24b190c6-kube-api-access-cl6p4\" (UniqueName: \"kubernetes.io/projected/7929d5a4-5bdb-4fac-82a0-514d24b190c6-kube-api-access-cl6p4\") pod \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") "
Oct 30 22:01:54.869260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.849407    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tls-certs\") pod \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") "
Oct 30 22:01:54.869260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.849713    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-dshm\") pod \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") "
Oct 30 22:01:54.869260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.849773    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tmp-dir\") pod \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") "
Oct 30 22:01:54.869260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.849834    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-model-cache\") pod \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\" (UID: \"7929d5a4-5bdb-4fac-82a0-514d24b190c6\") "
Oct 30 22:01:54.869260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.860996    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-model-cache" pod "7929d5a4-5bdb-4fac-82a0-514d24b190c6" (UID: "7929d5a4-5bdb-4fac-82a0-514d24b190c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:54.873611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.873560    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-home" pod "7929d5a4-5bdb-4fac-82a0-514d24b190c6" (UID: "7929d5a4-5bdb-4fac-82a0-514d24b190c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:54.922506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.915569    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tmp-dir" pod "7929d5a4-5bdb-4fac-82a0-514d24b190c6" (UID: "7929d5a4-5bdb-4fac-82a0-514d24b190c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:54.924646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.924611    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-dshm" pod "7929d5a4-5bdb-4fac-82a0-514d24b190c6" (UID: "7929d5a4-5bdb-4fac-82a0-514d24b190c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:01:54.924807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.924737    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tls-certs" pod "7929d5a4-5bdb-4fac-82a0-514d24b190c6" (UID: "7929d5a4-5bdb-4fac-82a0-514d24b190c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:54.925282 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.925255    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7929d5a4-5bdb-4fac-82a0-514d24b190c6-kube-api-access-cl6p4" pod "7929d5a4-5bdb-4fac-82a0-514d24b190c6" (UID: "7929d5a4-5bdb-4fac-82a0-514d24b190c6"). InnerVolumeSpecName "kube-api-access-cl6p4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:54.974494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.961318    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:54.974494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.961356    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:54.974494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.961373    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:54.974494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.961387    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl6p4\" (UniqueName: \"kubernetes.io/projected/7929d5a4-5bdb-4fac-82a0-514d24b190c6-kube-api-access-cl6p4\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:54.974494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.961401    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7929d5a4-5bdb-4fac-82a0-514d24b190c6-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:54.974494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:54.961416    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7929d5a4-5bdb-4fac-82a0-514d24b190c6-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:55.146631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.146539    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" event={"ID":"9eda7d94-5787-4a6d-97ce-bd2337998a88","Type":"ContainerStarted","Data":"56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81"}
Oct 30 22:01:55.146631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.146613    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" event={"ID":"9eda7d94-5787-4a6d-97ce-bd2337998a88","Type":"ContainerStarted","Data":"e537eda573b6d23c51d7fa221f96ed91975970be243d80c26145a5f59f29d5f8"}
Oct 30 22:01:55.148194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.148154    4282 generic.go:356] "Generic (PLEG): container finished" podID="7929d5a4-5bdb-4fac-82a0-514d24b190c6" containerID="01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975" exitCode=0
Oct 30 22:01:55.148283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.148239    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx" event={"ID":"7929d5a4-5bdb-4fac-82a0-514d24b190c6","Type":"ContainerDied","Data":"01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975"}
Oct 30 22:01:55.148283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.148275    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx" event={"ID":"7929d5a4-5bdb-4fac-82a0-514d24b190c6","Type":"ContainerDied","Data":"388bf15072402c1cf825a1c5105c8fdcb6bfc6ecc33a87350c210055371d8e1f"}
Oct 30 22:01:55.148366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.148295    4282 scope.go:122] "RemoveContainer" containerID="01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975"
Oct 30 22:01:55.148484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.148436    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"
Oct 30 22:01:55.203729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.201908    4282 scope.go:122] "RemoveContainer" containerID="01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975"
Oct 30 22:01:55.203729 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:01:55.202763    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975\": container with ID starting with 01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975 not found: ID does not exist" containerID="01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975"
Oct 30 22:01:55.203729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.202801    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975"} err="failed to get container status \"01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975\": rpc error: code = NotFound desc = could not find container \"01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975\": container with ID starting with 01ab524d2fbb3e44426cb91aee7e5bbd6bb6da9b521cd6ba1d86424a4bbf5975 not found: ID does not exist"
Oct 30 22:01:55.216553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.216499    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podStartSLOduration=2.216443229 podStartE2EDuration="2.216443229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:55.189594802 +0000 UTC m=+1333.847691028" watchObservedRunningTime="2026-10-30 22:01:55.216443229 +0000 UTC m=+1333.874539451"
Oct 30 22:01:55.217273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.217255    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"]
Oct 30 22:01:55.224126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.224099    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3654d5/ipp-route-3654d5-kserve-679c964f77-l5fvx"]
Oct 30 22:01:55.497493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:55.497422    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7929d5a4-5bdb-4fac-82a0-514d24b190c6" path="/var/lib/kubelet/pods/7929d5a4-5bdb-4fac-82a0-514d24b190c6/volumes"
Oct 30 22:01:56.080768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:56.080701    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:56.085014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:56.084990    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:56.198514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:56.198443    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:56.199582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:56.199554    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:01:59.022422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.022387    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"]
Oct 30 22:01:59.023075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.023058    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7929d5a4-5bdb-4fac-82a0-514d24b190c6" containerName="main"
Oct 30 22:01:59.023124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.023079    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7929d5a4-5bdb-4fac-82a0-514d24b190c6" containerName="main"
Oct 30 22:01:59.023266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.023245    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7929d5a4-5bdb-4fac-82a0-514d24b190c6" containerName="main"
Oct 30 22:01:59.028346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.028323    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:01:59.036696 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.036399    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bca8c9\""
Oct 30 22:01:59.037260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.037242    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bca8c9-dockercfg-zqddk\""
Oct 30 22:01:59.055653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.055625    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"]
Oct 30 22:01:59.067344 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.067312    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"]
Oct 30 22:01:59.072206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.072180    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:01:59.086022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.085996    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"]
Oct 30 22:01:59.220499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.217012    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l87q\" (UniqueName: \"kubernetes.io/projected/21d98e90-7393-4ad3-8411-1f2652c08ac2-kube-api-access-6l87q\") pod \"payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf\" (UID: \"21d98e90-7393-4ad3-8411-1f2652c08ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:01:59.220499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.217119    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n49m\" (UniqueName: \"kubernetes.io/projected/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-kube-api-access-7n49m\") pod \"payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb\" (UID: \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:01:59.220499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.217319    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb\" (UID: \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:01:59.220499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.217360    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d98e90-7393-4ad3-8411-1f2652c08ac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf\" (UID: \"21d98e90-7393-4ad3-8411-1f2652c08ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:01:59.325711 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.325297    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l87q\" (UniqueName: \"kubernetes.io/projected/21d98e90-7393-4ad3-8411-1f2652c08ac2-kube-api-access-6l87q\") pod \"payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf\" (UID: \"21d98e90-7393-4ad3-8411-1f2652c08ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:01:59.326077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.326012    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n49m\" (UniqueName: \"kubernetes.io/projected/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-kube-api-access-7n49m\") pod \"payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb\" (UID: \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:01:59.328098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.328075    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb\" (UID: \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:01:59.329204 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.329186    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d98e90-7393-4ad3-8411-1f2652c08ac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf\" (UID: \"21d98e90-7393-4ad3-8411-1f2652c08ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:01:59.330227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.329114    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb\" (UID: \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:01:59.330227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.330090    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d98e90-7393-4ad3-8411-1f2652c08ac2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf\" (UID: \"21d98e90-7393-4ad3-8411-1f2652c08ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:01:59.341793 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.340142    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l87q\" (UniqueName: \"kubernetes.io/projected/21d98e90-7393-4ad3-8411-1f2652c08ac2-kube-api-access-6l87q\") pod \"payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf\" (UID: \"21d98e90-7393-4ad3-8411-1f2652c08ac2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:01:59.345269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.344770    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:01:59.373994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.371849    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n49m\" (UniqueName: \"kubernetes.io/projected/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-kube-api-access-7n49m\") pod \"payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb\" (UID: \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:01:59.396228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:01:59.395974    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:02:00.249228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:00.249195    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"]
Oct 30 22:02:00.249919 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:02:00.249888    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d98e90_7393_4ad3_8411_1f2652c08ac2.slice/crio-e8001f8463154e5d1016c020058bfe97fdee08c965f1d8ea504f4a8eba696d12 WatchSource:0}: Error finding container e8001f8463154e5d1016c020058bfe97fdee08c965f1d8ea504f4a8eba696d12: Status 404 returned error can't find the container with id e8001f8463154e5d1016c020058bfe97fdee08c965f1d8ea504f4a8eba696d12
Oct 30 22:02:00.306427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:00.306403    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"]
Oct 30 22:02:00.307901 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:02:00.307872    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fa41ca_03bc_4817_a38c_ac1c9cc1d10d.slice/crio-9f9b4ddfb3e0118fdb66af203dbf9b53db5d11ed0acac6da8604da3246956f4a WatchSource:0}: Error finding container 9f9b4ddfb3e0118fdb66af203dbf9b53db5d11ed0acac6da8604da3246956f4a: Status 404 returned error can't find the container with id 9f9b4ddfb3e0118fdb66af203dbf9b53db5d11ed0acac6da8604da3246956f4a
Oct 30 22:02:00.356281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:00.356101    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf" event={"ID":"21d98e90-7393-4ad3-8411-1f2652c08ac2","Type":"ContainerStarted","Data":"95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a"}
Oct 30 22:02:00.356281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:00.356247    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf" event={"ID":"21d98e90-7393-4ad3-8411-1f2652c08ac2","Type":"ContainerStarted","Data":"e8001f8463154e5d1016c020058bfe97fdee08c965f1d8ea504f4a8eba696d12"}
Oct 30 22:02:00.356586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:00.356451    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:02:00.358458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:00.358431    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb" event={"ID":"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d","Type":"ContainerStarted","Data":"9f9b4ddfb3e0118fdb66af203dbf9b53db5d11ed0acac6da8604da3246956f4a"}
Oct 30 22:02:00.381929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:00.381850    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf" podStartSLOduration=2.3818298 podStartE2EDuration="2.3818298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:00.380648548 +0000 UTC m=+1339.038744770" watchObservedRunningTime="2026-10-30 22:02:00.3818298 +0000 UTC m=+1339.039926024"
Oct 30 22:02:00.990198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:00.990168    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:02:01.086977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.086881    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-credential-socket\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.086977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.086944    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istiod-ca-cert\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.087221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.086989    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-token\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-token\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.087221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087048    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-certs\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.087221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087154    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-podinfo\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.087221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087168    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-credential-socket" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:01.087221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087197    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-data\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.087536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087247    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-kube-api-access-6rbpp\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-kube-api-access-6rbpp\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.087536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087386    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-certs" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:01.087536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087408    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-socket\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.087536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087506    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-envoy\") pod \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\" (UID: \"519c0cc2-aeef-4a66-b158-f6e703e5d0da\") "
Oct 30 22:02:01.087536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087520    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-data" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:01.087822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087540    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istiod-ca-cert" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:01.087822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.087696    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-socket" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:01.088411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.088386    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.088561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.088413    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.088561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.088429    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.088561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.088444    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.088561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.088483    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.089867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.089839    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-kube-api-access-6rbpp" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "kube-api-access-6rbpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:01.089958 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.089922    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-envoy" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:01.090153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.090130    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-token" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:01.090299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.090284    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-podinfo" pod "519c0cc2-aeef-4a66-b158-f6e703e5d0da" (UID: "519c0cc2-aeef-4a66-b158-f6e703e5d0da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:02:01.189928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.189890    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.189928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.189931    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.190177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.189946    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519c0cc2-aeef-4a66-b158-f6e703e5d0da-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.190177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.189961    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rbpp\" (UniqueName: \"kubernetes.io/projected/519c0cc2-aeef-4a66-b158-f6e703e5d0da-kube-api-access-6rbpp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:01.368334 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.368260    4282 generic.go:356] "Generic (PLEG): container finished" podID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerID="95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183" exitCode=0
Oct 30 22:02:01.368807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.368337    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"
Oct 30 22:02:01.368807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.368345    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" event={"ID":"519c0cc2-aeef-4a66-b158-f6e703e5d0da","Type":"ContainerDied","Data":"95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183"}
Oct 30 22:02:01.368807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.368413    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns" event={"ID":"519c0cc2-aeef-4a66-b158-f6e703e5d0da","Type":"ContainerDied","Data":"46a030ba06c535341654056986462534cfcf776c97d997b10130cd31e9c75250"}
Oct 30 22:02:01.368807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.368446    4282 scope.go:122] "RemoveContainer" containerID="95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183"
Oct 30 22:02:01.370314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.370287    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb" event={"ID":"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d","Type":"ContainerStarted","Data":"421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87"}
Oct 30 22:02:01.380114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.380091    4282 scope.go:122] "RemoveContainer" containerID="9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002"
Oct 30 22:02:01.389654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.389632    4282 scope.go:122] "RemoveContainer" containerID="95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183"
Oct 30 22:02:01.389942 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:01.389913    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183\": container with ID starting with 95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183 not found: ID does not exist" containerID="95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183"
Oct 30 22:02:01.389999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.389955    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183"} err="failed to get container status \"95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183\": rpc error: code = NotFound desc = could not find container \"95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183\": container with ID starting with 95e8bf6362274e7ea65f0a8eb9babd52debbec8839a45f0145e291c3284a6183 not found: ID does not exist"
Oct 30 22:02:01.389999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.389977    4282 scope.go:122] "RemoveContainer" containerID="9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002"
Oct 30 22:02:01.390245 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:01.390227    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002\": container with ID starting with 9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002 not found: ID does not exist" containerID="9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002"
Oct 30 22:02:01.390295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.390254    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002"} err="failed to get container status \"9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002\": rpc error: code = NotFound desc = could not find container \"9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002\": container with ID starting with 9e7438ca564bf1ee5b06f5a4b53bba324ae0e8811f12ae5863fc0cac17e15002 not found: ID does not exist"
Oct 30 22:02:01.403361 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.403317    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb" podStartSLOduration=2.403306062 podStartE2EDuration="2.403306062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:01.402386251 +0000 UTC m=+1340.060482474" watchObservedRunningTime="2026-10-30 22:02:01.403306062 +0000 UTC m=+1340.061402292"
Oct 30 22:02:01.423688 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.423667    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"]
Oct 30 22:02:01.431272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.431251    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ecffad-openshift-default-7db8446cb9-mmwns"]
Oct 30 22:02:01.466433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:01.466405    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" path="/var/lib/kubelet/pods/519c0cc2-aeef-4a66-b158-f6e703e5d0da/volumes"
Oct 30 22:02:02.380228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:02.380203    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:02:03.389686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:03.389647    4282 generic.go:356] "Generic (PLEG): container finished" podID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerID="8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2" exitCode=0
Oct 30 22:02:03.390170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:03.389715    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" event={"ID":"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d","Type":"ContainerDied","Data":"8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2"}
Oct 30 22:02:03.390170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:03.390095    4282 scope.go:122] "RemoveContainer" containerID="8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2"
Oct 30 22:02:03.391560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:03.391528    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:03.391686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:03.391607    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:03.391686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:03.391639    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:04.079697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.079655    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:02:04.399562 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.399454    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" event={"ID":"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d","Type":"ContainerStarted","Data":"47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228"}
Oct 30 22:02:04.427776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.427740    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"]
Oct 30 22:02:04.428424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.428404    4282 cpu_manager.go:414] "containerMap: removing container" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy"
Oct 30 22:02:04.428569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.428425    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy"
Oct 30 22:02:04.428569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.428441    4282 cpu_manager.go:414] "containerMap: removing container" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy"
Oct 30 22:02:04.428569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.428448    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy"
Oct 30 22:02:04.428723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.428682    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy"
Oct 30 22:02:04.428723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.428698    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="519c0cc2-aeef-4a66-b158-f6e703e5d0da" containerName="istio-proxy"
Oct 30 22:02:04.432299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.432278    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:04.435994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.435974    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4cae46\""
Oct 30 22:02:04.489187 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.489151    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"]
Oct 30 22:02:04.490957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.490931    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"]
Oct 30 22:02:04.495305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.495283    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:04.509905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.509880    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4cae46-dockercfg-4ssjb\""
Oct 30 22:02:04.514553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.514528    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"]
Oct 30 22:02:04.663548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.624830    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e94194-d308-4852-b02f-ddb391f11127-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5\" (UID: \"a8e94194-d308-4852-b02f-ddb391f11127\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:04.664158 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.664122    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggd7s\" (UniqueName: \"kubernetes.io/projected/a8e94194-d308-4852-b02f-ddb391f11127-kube-api-access-ggd7s\") pod \"payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5\" (UID: \"a8e94194-d308-4852-b02f-ddb391f11127\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:04.797767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.797728    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggd7s\" (UniqueName: \"kubernetes.io/projected/a8e94194-d308-4852-b02f-ddb391f11127-kube-api-access-ggd7s\") pod \"payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5\" (UID: \"a8e94194-d308-4852-b02f-ddb391f11127\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:04.844162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.822729    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e94194-d308-4852-b02f-ddb391f11127-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5\" (UID: \"a8e94194-d308-4852-b02f-ddb391f11127\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:04.844162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.824135    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e94194-d308-4852-b02f-ddb391f11127-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5\" (UID: \"a8e94194-d308-4852-b02f-ddb391f11127\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:04.929908 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.929793    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9sf\" (UniqueName: \"kubernetes.io/projected/7600b93e-60ad-4ac0-bc34-bdcc90047403-kube-api-access-mc9sf\") pod \"payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc\" (UID: \"7600b93e-60ad-4ac0-bc34-bdcc90047403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:04.930233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.930211    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7600b93e-60ad-4ac0-bc34-bdcc90047403-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc\" (UID: \"7600b93e-60ad-4ac0-bc34-bdcc90047403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:04.994515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:04.978637    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggd7s\" (UniqueName: \"kubernetes.io/projected/a8e94194-d308-4852-b02f-ddb391f11127-kube-api-access-ggd7s\") pod \"payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5\" (UID: \"a8e94194-d308-4852-b02f-ddb391f11127\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:05.014734 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:05.014689    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:02:05.050943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.050903    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:05.064951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.064906    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9sf\" (UniqueName: \"kubernetes.io/projected/7600b93e-60ad-4ac0-bc34-bdcc90047403-kube-api-access-mc9sf\") pod \"payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc\" (UID: \"7600b93e-60ad-4ac0-bc34-bdcc90047403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:05.065424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.065393    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7600b93e-60ad-4ac0-bc34-bdcc90047403-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc\" (UID: \"7600b93e-60ad-4ac0-bc34-bdcc90047403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:05.077983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.075143    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7600b93e-60ad-4ac0-bc34-bdcc90047403-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc\" (UID: \"7600b93e-60ad-4ac0-bc34-bdcc90047403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:05.110514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.104314    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9sf\" (UniqueName: \"kubernetes.io/projected/7600b93e-60ad-4ac0-bc34-bdcc90047403-kube-api-access-mc9sf\") pod \"payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc\" (UID: \"7600b93e-60ad-4ac0-bc34-bdcc90047403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:05.115789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.115755    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:05.314751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.314726    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:02:05.315755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.315728    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:02:05.315895 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.315783    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:02:05.508552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.508525    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"]
Oct 30 22:02:05.511308 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:02:05.511269    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e94194_d308_4852_b02f_ddb391f11127.slice/crio-f74e032071766ab77eaeacc7e242f4d6ea4614cc1d2041e8e0110cfb21f78141 WatchSource:0}: Error finding container f74e032071766ab77eaeacc7e242f4d6ea4614cc1d2041e8e0110cfb21f78141: Status 404 returned error can't find the container with id f74e032071766ab77eaeacc7e242f4d6ea4614cc1d2041e8e0110cfb21f78141
Oct 30 22:02:05.515102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.515079    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"]
Oct 30 22:02:05.515214 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:02:05.515195    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7600b93e_60ad_4ac0_bc34_bdcc90047403.slice/crio-e7809be16dedfca241223261440514bc82e14737af820b485993776f8c87e096 WatchSource:0}: Error finding container e7809be16dedfca241223261440514bc82e14737af820b485993776f8c87e096: Status 404 returned error can't find the container with id e7809be16dedfca241223261440514bc82e14737af820b485993776f8c87e096
Oct 30 22:02:05.577766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.577727    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"]
Oct 30 22:02:05.591909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:05.591882    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"]
Oct 30 22:02:06.314872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.314838    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:02:06.315181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.315150    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:02:06.428275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.428240    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5" event={"ID":"a8e94194-d308-4852-b02f-ddb391f11127","Type":"ContainerStarted","Data":"87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c"}
Oct 30 22:02:06.428915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.428890    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5" event={"ID":"a8e94194-d308-4852-b02f-ddb391f11127","Type":"ContainerStarted","Data":"f74e032071766ab77eaeacc7e242f4d6ea4614cc1d2041e8e0110cfb21f78141"}
Oct 30 22:02:06.429124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.429109    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:06.431754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.431700    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc" event={"ID":"7600b93e-60ad-4ac0-bc34-bdcc90047403","Type":"ContainerStarted","Data":"31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e"}
Oct 30 22:02:06.431924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.431908    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc" event={"ID":"7600b93e-60ad-4ac0-bc34-bdcc90047403","Type":"ContainerStarted","Data":"e7809be16dedfca241223261440514bc82e14737af820b485993776f8c87e096"}
Oct 30 22:02:06.432055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.432043    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:06.478575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.478513    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5" podStartSLOduration=2.478494335 podStartE2EDuration="2.478494335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:06.478100914 +0000 UTC m=+1345.136197139" watchObservedRunningTime="2026-10-30 22:02:06.478494335 +0000 UTC m=+1345.136590562"
Oct 30 22:02:06.499760 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:06.499703    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc" podStartSLOduration=2.499689991 podStartE2EDuration="2.499689991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:06.498125205 +0000 UTC m=+1345.156221428" watchObservedRunningTime="2026-10-30 22:02:06.499689991 +0000 UTC m=+1345.157786235"
Oct 30 22:02:07.314643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:07.314602    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:02:07.315132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:07.314659    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:02:08.314674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.314646    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:02:08.315091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.314695    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:02:08.918592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.918562    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"]
Oct 30 22:02:08.923839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.923814    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:08.939036 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.938990    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bca8c9-serving-cert\""
Oct 30 22:02:08.941905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.941879    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"]
Oct 30 22:02:08.997552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.997523    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5gx\" (UniqueName: \"kubernetes.io/projected/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-kube-api-access-8h5gx\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:08.997739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.997709    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:08.997808 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.997752    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:08.997896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:08.997874    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.102496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.101552    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.102496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.101636    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.102496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.101750    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.102496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.101890    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5gx\" (UniqueName: \"kubernetes.io/projected/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-kube-api-access-8h5gx\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.106490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.103791    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.106490 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:09.103910    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-serving-cert: secret "maas-api-e2e-praxis-b-bca8c9-serving-cert" not found
Oct 30 22:02:09.106490 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:09.103972    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls podName:f89a890d-368e-419c-8c6c-7ad5e0b78fe0 nodeName:}" failed. No retries permitted until 2026-10-30 22:02:09.603952646 +0000 UTC m=+1348.262048864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls") pod "maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l" (UID: "f89a890d-368e-419c-8c6c-7ad5e0b78fe0") : secret "maas-api-e2e-praxis-b-bca8c9-serving-cert" not found
Oct 30 22:02:09.163167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.163132    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.164077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.164055    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5gx\" (UniqueName: \"kubernetes.io/projected/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-kube-api-access-8h5gx\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.314816 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.314737    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Oct 30 22:02:09.314816 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.314794    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Oct 30 22:02:09.623548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:09.617725    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:09.623548 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:09.618053    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-serving-cert: secret "maas-api-e2e-praxis-b-bca8c9-serving-cert" not found
Oct 30 22:02:09.623548 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:09.618125    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls podName:f89a890d-368e-419c-8c6c-7ad5e0b78fe0 nodeName:}" failed. No retries permitted until 2026-10-30 22:02:10.618112211 +0000 UTC m=+1349.276208411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls") pod "maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l" (UID: "f89a890d-368e-419c-8c6c-7ad5e0b78fe0") : secret "maas-api-e2e-praxis-b-bca8c9-serving-cert" not found
Oct 30 22:02:10.319725 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:10.319694    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:02:10.320185 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:10.320173    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:02:10.320951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:10.320933    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:02:10.578188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:10.578103    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk" podUID="9d07ff68-ceb1-426a-9feb-d3d5ec90029d" containerName="payload-processing" containerID="cri-o://5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423" gracePeriod=25
Oct 30 22:02:10.593275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:10.593241    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn" podUID="9045806c-c579-4ffe-838a-f31d6960b166" containerName="payload-pre-processing" containerID="cri-o://0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd" gracePeriod=25
Oct 30 22:02:10.674501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:10.672111    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:10.699089 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:10.699053    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:10.741227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:10.740697    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:11.025356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.025335    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:02:11.027950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.027927    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:02:11.079963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.079933    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9045806c-c579-4ffe-838a-f31d6960b166-kube-api-access-hhzjb\" (UniqueName: \"kubernetes.io/projected/9045806c-c579-4ffe-838a-f31d6960b166-kube-api-access-hhzjb\") pod \"9045806c-c579-4ffe-838a-f31d6960b166\" (UID: \"9045806c-c579-4ffe-838a-f31d6960b166\") "
Oct 30 22:02:11.080156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.080069    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9045806c-c579-4ffe-838a-f31d6960b166-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9045806c-c579-4ffe-838a-f31d6960b166-plugins-config-volume\") pod \"9045806c-c579-4ffe-838a-f31d6960b166\" (UID: \"9045806c-c579-4ffe-838a-f31d6960b166\") "
Oct 30 22:02:11.080156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.080127    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-kube-api-access-z44jd\" (UniqueName: \"kubernetes.io/projected/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-kube-api-access-z44jd\") pod \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\" (UID: \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\") "
Oct 30 22:02:11.080280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.080261    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-plugins-config-volume\") pod \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\" (UID: \"9d07ff68-ceb1-426a-9feb-d3d5ec90029d\") "
Oct 30 22:02:11.082292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.082266    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-plugins-config-volume" pod "9d07ff68-ceb1-426a-9feb-d3d5ec90029d" (UID: "9d07ff68-ceb1-426a-9feb-d3d5ec90029d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:11.082639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.082617    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9045806c-c579-4ffe-838a-f31d6960b166-plugins-config-volume" pod "9045806c-c579-4ffe-838a-f31d6960b166" (UID: "9045806c-c579-4ffe-838a-f31d6960b166"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:11.083781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.083752    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9045806c-c579-4ffe-838a-f31d6960b166-kube-api-access-hhzjb" pod "9045806c-c579-4ffe-838a-f31d6960b166" (UID: "9045806c-c579-4ffe-838a-f31d6960b166"). InnerVolumeSpecName "kube-api-access-hhzjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:11.084737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.084712    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-kube-api-access-z44jd" pod "9d07ff68-ceb1-426a-9feb-d3d5ec90029d" (UID: "9d07ff68-ceb1-426a-9feb-d3d5ec90029d"). InnerVolumeSpecName "kube-api-access-z44jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:11.086648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.086620    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"]
Oct 30 22:02:11.089939 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:02:11.089901    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89a890d_368e_419c_8c6c_7ad5e0b78fe0.slice/crio-c33b188ee9f6634c7d1994b606334f5887f48aa282bb39217a11ea9375de3b09 WatchSource:0}: Error finding container c33b188ee9f6634c7d1994b606334f5887f48aa282bb39217a11ea9375de3b09: Status 404 returned error can't find the container with id c33b188ee9f6634c7d1994b606334f5887f48aa282bb39217a11ea9375de3b09
Oct 30 22:02:11.183023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.182995    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9045806c-c579-4ffe-838a-f31d6960b166-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.183023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.183020    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z44jd\" (UniqueName: \"kubernetes.io/projected/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-kube-api-access-z44jd\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.183217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.183036    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d07ff68-ceb1-426a-9feb-d3d5ec90029d-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.183217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.183048    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzjb\" (UniqueName: \"kubernetes.io/projected/9045806c-c579-4ffe-838a-f31d6960b166-kube-api-access-hhzjb\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:11.398496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.393709    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:02:11.516677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.516640    4282 generic.go:356] "Generic (PLEG): container finished" podID="9045806c-c579-4ffe-838a-f31d6960b166" containerID="0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd" exitCode=0
Oct 30 22:02:11.516854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.516713    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"
Oct 30 22:02:11.516854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.516718    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn" event={"ID":"9045806c-c579-4ffe-838a-f31d6960b166","Type":"ContainerDied","Data":"0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd"}
Oct 30 22:02:11.516854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.516789    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn" event={"ID":"9045806c-c579-4ffe-838a-f31d6960b166","Type":"ContainerDied","Data":"38f049d73d1c911c9ecfd0d6f9f530ac35ea7cba328725a582481f9b5b70e1d8"}
Oct 30 22:02:11.516854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.516810    4282 scope.go:122] "RemoveContainer" containerID="0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd"
Oct 30 22:02:11.518703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.518676    4282 generic.go:356] "Generic (PLEG): container finished" podID="9d07ff68-ceb1-426a-9feb-d3d5ec90029d" containerID="5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423" exitCode=0
Oct 30 22:02:11.518936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.518917    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"
Oct 30 22:02:11.519535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.519449    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk" event={"ID":"9d07ff68-ceb1-426a-9feb-d3d5ec90029d","Type":"ContainerDied","Data":"5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423"}
Oct 30 22:02:11.519648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.519568    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk" event={"ID":"9d07ff68-ceb1-426a-9feb-d3d5ec90029d","Type":"ContainerDied","Data":"0b5ad4f15928c6d2a49f576d6c4a29de0c757e013f5a6364d8e5d0f1eefc5b1d"}
Oct 30 22:02:11.521277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.521258    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l" event={"ID":"f89a890d-368e-419c-8c6c-7ad5e0b78fe0","Type":"ContainerStarted","Data":"c33b188ee9f6634c7d1994b606334f5887f48aa282bb39217a11ea9375de3b09"}
Oct 30 22:02:11.545656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.545627    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"]
Oct 30 22:02:11.546827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.546802    4282 scope.go:122] "RemoveContainer" containerID="0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd"
Oct 30 22:02:11.547189 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:11.547164    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd\": container with ID starting with 0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd not found: ID does not exist" containerID="0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd"
Oct 30 22:02:11.547292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.547194    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd"} err="failed to get container status \"0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd\": rpc error: code = NotFound desc = could not find container \"0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd\": container with ID starting with 0da28ce14e20138cccc879168bde0ff0199d1679204ab7246d6f18d6ead214fd not found: ID does not exist"
Oct 30 22:02:11.547292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.547212    4282 scope.go:122] "RemoveContainer" containerID="5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423"
Oct 30 22:02:11.550206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.550182    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-3724d9-6897cdbb75-rsnwn"]
Oct 30 22:02:11.559584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.559566    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"]
Oct 30 22:02:11.560037 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.560017    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-3724d9-57b5bf9479-ccdhk"]
Oct 30 22:02:11.566814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.566799    4282 scope.go:122] "RemoveContainer" containerID="5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423"
Oct 30 22:02:11.567112 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:11.567088    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423\": container with ID starting with 5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423 not found: ID does not exist" containerID="5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423"
Oct 30 22:02:11.567174 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:11.567122    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423"} err="failed to get container status \"5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423\": rpc error: code = NotFound desc = could not find container \"5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423\": container with ID starting with 5e465555a2c2cb2c175c7e318bdae0225e9b12bd7262af7de144e24417e97423 not found: ID does not exist"
Oct 30 22:02:12.531267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:12.531229    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l" event={"ID":"f89a890d-368e-419c-8c6c-7ad5e0b78fe0","Type":"ContainerStarted","Data":"1a5ee8d361b97229406623bc7ef4384bde773a7b14b4ebc2eeb12b9d69892872"}
Oct 30 22:02:12.531740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:12.531424    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:12.551841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:12.551797    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l" podStartSLOduration=4.091076426 podStartE2EDuration="4.551785256s" totalImagesPullingTime="460.70883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:12.549531494 +0000 UTC m=+1351.207627717" watchObservedRunningTime="2026-10-30 22:02:12.551785256 +0000 UTC m=+1351.209881479"
Oct 30 22:02:13.393680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.393655    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:02:13.467339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.467310    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9045806c-c579-4ffe-838a-f31d6960b166" path="/var/lib/kubelet/pods/9045806c-c579-4ffe-838a-f31d6960b166/volumes"
Oct 30 22:02:13.467812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.467797    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d07ff68-ceb1-426a-9feb-d3d5ec90029d" path="/var/lib/kubelet/pods/9d07ff68-ceb1-426a-9feb-d3d5ec90029d/volumes"
Oct 30 22:02:13.521350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.521323    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"]
Oct 30 22:02:13.522248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.522222    4282 cpu_manager.go:414] "containerMap: removing container" podUID="9d07ff68-ceb1-426a-9feb-d3d5ec90029d" containerName="payload-processing"
Oct 30 22:02:13.522248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.522248    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d07ff68-ceb1-426a-9feb-d3d5ec90029d" containerName="payload-processing"
Oct 30 22:02:13.522454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.522266    4282 cpu_manager.go:414] "containerMap: removing container" podUID="9045806c-c579-4ffe-838a-f31d6960b166" containerName="payload-pre-processing"
Oct 30 22:02:13.522454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.522273    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9045806c-c579-4ffe-838a-f31d6960b166" containerName="payload-pre-processing"
Oct 30 22:02:13.522593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.522492    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d07ff68-ceb1-426a-9feb-d3d5ec90029d" containerName="payload-processing"
Oct 30 22:02:13.522593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.522509    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9045806c-c579-4ffe-838a-f31d6960b166" containerName="payload-pre-processing"
Oct 30 22:02:13.527572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.527553    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.534002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.533911    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4cae46-serving-cert\""
Oct 30 22:02:13.548836 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.548807    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"]
Oct 30 22:02:13.661550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.660197    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.661550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.660285    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwszf\" (UniqueName: \"kubernetes.io/projected/ea05b385-c1b2-4020-839b-843de45e95bf-kube-api-access-lwszf\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.661550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.660356    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.661550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.660711    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea05b385-c1b2-4020-839b-843de45e95bf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.761895 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.761858    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea05b385-c1b2-4020-839b-843de45e95bf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.762075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.761977    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.762075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.762015    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwszf\" (UniqueName: \"kubernetes.io/projected/ea05b385-c1b2-4020-839b-843de45e95bf-kube-api-access-lwszf\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.762075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.762062    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.763603 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.763445    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea05b385-c1b2-4020-839b-843de45e95bf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.766075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.766049    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.766570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.766547    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-metrics-certs\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.774328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.774304    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwszf\" (UniqueName: \"kubernetes.io/projected/ea05b385-c1b2-4020-839b-843de45e95bf-kube-api-access-lwszf\") pod \"maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:13.854309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:13.854284    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:14.261492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.258793    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"]
Oct 30 22:02:14.261492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.260340    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb" podUID="c0f801ad-9939-400d-b564-c013416f3f6e" containerName="maas-api" containerID="cri-o://0389e75aadd85931dbda4bc7de0cd8f11e2c08de71798f46bb78b4a9ebbab6d2" gracePeriod=30
Oct 30 22:02:14.318987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.317813    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"]
Oct 30 22:02:14.353532 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:02:14.353497    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea05b385_c1b2_4020_839b_843de45e95bf.slice/crio-2ed0a1014cd2f65b5fb2b134ebff0c4d28ed781e8f944e68016dd30c3426d8ff WatchSource:0}: Error finding container 2ed0a1014cd2f65b5fb2b134ebff0c4d28ed781e8f944e68016dd30c3426d8ff: Status 404 returned error can't find the container with id 2ed0a1014cd2f65b5fb2b134ebff0c4d28ed781e8f944e68016dd30c3426d8ff
Oct 30 22:02:14.805859 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.805388    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2" event={"ID":"ea05b385-c1b2-4020-839b-843de45e95bf","Type":"ContainerStarted","Data":"2ed0a1014cd2f65b5fb2b134ebff0c4d28ed781e8f944e68016dd30c3426d8ff"}
Oct 30 22:02:14.827191 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.825498    4282 generic.go:356] "Generic (PLEG): container finished" podID="c0f801ad-9939-400d-b564-c013416f3f6e" containerID="0389e75aadd85931dbda4bc7de0cd8f11e2c08de71798f46bb78b4a9ebbab6d2" exitCode=0
Oct 30 22:02:14.827191 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.825556    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb" event={"ID":"c0f801ad-9939-400d-b564-c013416f3f6e","Type":"ContainerDied","Data":"0389e75aadd85931dbda4bc7de0cd8f11e2c08de71798f46bb78b4a9ebbab6d2"}
Oct 30 22:02:14.879374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.879235    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:02:14.894665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.894623    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"]
Oct 30 22:02:14.898930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.898905    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"]
Oct 30 22:02:14.936876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.927947    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls\") pod \"c0f801ad-9939-400d-b564-c013416f3f6e\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") "
Oct 30 22:02:14.936876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.928034    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f801ad-9939-400d-b564-c013416f3f6e-kube-api-access-449h5\" (UniqueName: \"kubernetes.io/projected/c0f801ad-9939-400d-b564-c013416f3f6e-kube-api-access-449h5\") pod \"c0f801ad-9939-400d-b564-c013416f3f6e\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") "
Oct 30 22:02:14.936876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.928140    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0f801ad-9939-400d-b564-c013416f3f6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f801ad-9939-400d-b564-c013416f3f6e-openshift-service-ca\") pod \"c0f801ad-9939-400d-b564-c013416f3f6e\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") "
Oct 30 22:02:14.936876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.928304    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-metrics-certs\") pod \"c0f801ad-9939-400d-b564-c013416f3f6e\" (UID: \"c0f801ad-9939-400d-b564-c013416f3f6e\") "
Oct 30 22:02:14.936876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.931867    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0f801ad-9939-400d-b564-c013416f3f6e-openshift-service-ca" pod "c0f801ad-9939-400d-b564-c013416f3f6e" (UID: "c0f801ad-9939-400d-b564-c013416f3f6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:14.937509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.937481    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f801ad-9939-400d-b564-c013416f3f6e-kube-api-access-449h5" pod "c0f801ad-9939-400d-b564-c013416f3f6e" (UID: "c0f801ad-9939-400d-b564-c013416f3f6e"). InnerVolumeSpecName "kube-api-access-449h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:14.937915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.937875    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-metrics-certs" pod "c0f801ad-9939-400d-b564-c013416f3f6e" (UID: "c0f801ad-9939-400d-b564-c013416f3f6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:14.939000 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:14.938971    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls" pod "c0f801ad-9939-400d-b564-c013416f3f6e" (UID: "c0f801ad-9939-400d-b564-c013416f3f6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:15.031106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.031077    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0f801ad-9939-400d-b564-c013416f3f6e-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:15.031106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.031104    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:15.031106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.031114    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0f801ad-9939-400d-b564-c013416f3f6e-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:15.031368 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.031123    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-449h5\" (UniqueName: \"kubernetes.io/projected/c0f801ad-9939-400d-b564-c013416f3f6e-kube-api-access-449h5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:15.848545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.848507    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2" event={"ID":"ea05b385-c1b2-4020-839b-843de45e95bf","Type":"ContainerStarted","Data":"b0744e3222314b4c3eb252c51c674da3fa670a811facaf762b003dee9d5c9a65"}
Oct 30 22:02:15.849002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.848722    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:15.850345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.850327    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"
Oct 30 22:02:15.850596 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.850329    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb" event={"ID":"c0f801ad-9939-400d-b564-c013416f3f6e","Type":"ContainerDied","Data":"63f77dddb19107327fc897e8d3b589930b05d8d3b0aa15269edc91e892130c1c"}
Oct 30 22:02:15.850813 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.850797    4282 scope.go:122] "RemoveContainer" containerID="0389e75aadd85931dbda4bc7de0cd8f11e2c08de71798f46bb78b4a9ebbab6d2"
Oct 30 22:02:15.890493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.882128    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2" podStartSLOduration=2.332376694 podStartE2EDuration="2.882112518s" totalImagesPullingTime="549.735824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:15.877152688 +0000 UTC m=+1354.535248911" watchObservedRunningTime="2026-10-30 22:02:15.882112518 +0000 UTC m=+1354.540208742"
Oct 30 22:02:15.906527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.906458    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"]
Oct 30 22:02:15.910029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:15.910006    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3724d9-bfd7b4c65-qz6pb"]
Oct 30 22:02:16.034962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:16.034935    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:02:17.465158 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:17.465125    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f801ad-9939-400d-b564-c013416f3f6e" path="/var/lib/kubelet/pods/c0f801ad-9939-400d-b564-c013416f3f6e/volumes"
Oct 30 22:02:17.481671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:17.481646    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:02:18.487926 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:18.487902    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:02:18.550235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:18.550211    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:02:19.896117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:19.896063    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9" podUID="174221eb-a62d-49a2-8ffc-e209628e641c" containerName="payload-processing" containerID="cri-o://7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0" gracePeriod=25
Oct 30 22:02:19.899725 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:19.899690    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh" podUID="2445cdba-b53a-4f2f-898c-7c0b837e8571" containerName="payload-pre-processing" containerID="cri-o://c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c" gracePeriod=25
Oct 30 22:02:20.192132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.192104    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:02:20.208654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.208635    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:02:20.255724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.255695    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2445cdba-b53a-4f2f-898c-7c0b837e8571-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2445cdba-b53a-4f2f-898c-7c0b837e8571-plugins-config-volume\") pod \"2445cdba-b53a-4f2f-898c-7c0b837e8571\" (UID: \"2445cdba-b53a-4f2f-898c-7c0b837e8571\") "
Oct 30 22:02:20.255892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.255743    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2445cdba-b53a-4f2f-898c-7c0b837e8571-kube-api-access-wtgrx\" (UniqueName: \"kubernetes.io/projected/2445cdba-b53a-4f2f-898c-7c0b837e8571-kube-api-access-wtgrx\") pod \"2445cdba-b53a-4f2f-898c-7c0b837e8571\" (UID: \"2445cdba-b53a-4f2f-898c-7c0b837e8571\") "
Oct 30 22:02:20.255892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.255811    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174221eb-a62d-49a2-8ffc-e209628e641c-kube-api-access-vcjdj\" (UniqueName: \"kubernetes.io/projected/174221eb-a62d-49a2-8ffc-e209628e641c-kube-api-access-vcjdj\") pod \"174221eb-a62d-49a2-8ffc-e209628e641c\" (UID: \"174221eb-a62d-49a2-8ffc-e209628e641c\") "
Oct 30 22:02:20.256066 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.256044    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/174221eb-a62d-49a2-8ffc-e209628e641c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174221eb-a62d-49a2-8ffc-e209628e641c-plugins-config-volume\") pod \"174221eb-a62d-49a2-8ffc-e209628e641c\" (UID: \"174221eb-a62d-49a2-8ffc-e209628e641c\") "
Oct 30 22:02:20.256137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.256097    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2445cdba-b53a-4f2f-898c-7c0b837e8571-plugins-config-volume" pod "2445cdba-b53a-4f2f-898c-7c0b837e8571" (UID: "2445cdba-b53a-4f2f-898c-7c0b837e8571"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:20.256483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.256419    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174221eb-a62d-49a2-8ffc-e209628e641c-plugins-config-volume" pod "174221eb-a62d-49a2-8ffc-e209628e641c" (UID: "174221eb-a62d-49a2-8ffc-e209628e641c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:20.256778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.256729    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2445cdba-b53a-4f2f-898c-7c0b837e8571-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:20.256778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.256746    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/174221eb-a62d-49a2-8ffc-e209628e641c-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:20.258116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.258097    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174221eb-a62d-49a2-8ffc-e209628e641c-kube-api-access-vcjdj" pod "174221eb-a62d-49a2-8ffc-e209628e641c" (UID: "174221eb-a62d-49a2-8ffc-e209628e641c"). InnerVolumeSpecName "kube-api-access-vcjdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:20.258202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.258137    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2445cdba-b53a-4f2f-898c-7c0b837e8571-kube-api-access-wtgrx" pod "2445cdba-b53a-4f2f-898c-7c0b837e8571" (UID: "2445cdba-b53a-4f2f-898c-7c0b837e8571"). InnerVolumeSpecName "kube-api-access-wtgrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:20.357644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.357615    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtgrx\" (UniqueName: \"kubernetes.io/projected/2445cdba-b53a-4f2f-898c-7c0b837e8571-kube-api-access-wtgrx\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:20.357644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.357641    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcjdj\" (UniqueName: \"kubernetes.io/projected/174221eb-a62d-49a2-8ffc-e209628e641c-kube-api-access-vcjdj\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:20.931948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.931916    4282 generic.go:356] "Generic (PLEG): container finished" podID="2445cdba-b53a-4f2f-898c-7c0b837e8571" containerID="c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c" exitCode=0
Oct 30 22:02:20.932384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.931977    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"
Oct 30 22:02:20.932384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.932009    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh" event={"ID":"2445cdba-b53a-4f2f-898c-7c0b837e8571","Type":"ContainerDied","Data":"c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c"}
Oct 30 22:02:20.932384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.932066    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh" event={"ID":"2445cdba-b53a-4f2f-898c-7c0b837e8571","Type":"ContainerDied","Data":"1f1bed1cbfcbae095756d3e3e35829b28c6f992a0cf4f8ee4b3e1f0edb20eb55"}
Oct 30 22:02:20.932384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.932095    4282 scope.go:122] "RemoveContainer" containerID="c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c"
Oct 30 22:02:20.933716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.933692    4282 generic.go:356] "Generic (PLEG): container finished" podID="174221eb-a62d-49a2-8ffc-e209628e641c" containerID="7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0" exitCode=0
Oct 30 22:02:20.934044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.933738    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"
Oct 30 22:02:20.934044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.933750    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9" event={"ID":"174221eb-a62d-49a2-8ffc-e209628e641c","Type":"ContainerDied","Data":"7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0"}
Oct 30 22:02:20.934044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.933803    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9" event={"ID":"174221eb-a62d-49a2-8ffc-e209628e641c","Type":"ContainerDied","Data":"d30e6988421319f797098934e037991a2526c34cc56ee7fef215432a0f6ad688"}
Oct 30 22:02:20.944325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.944307    4282 scope.go:122] "RemoveContainer" containerID="c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c"
Oct 30 22:02:20.944826 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:20.944807    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c\": container with ID starting with c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c not found: ID does not exist" containerID="c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c"
Oct 30 22:02:20.944890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.944834    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c"} err="failed to get container status \"c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c\": rpc error: code = NotFound desc = could not find container \"c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c\": container with ID starting with c096b81ec8c481cd526d7d0513e4e7602ad4d577e1f074bb0c0f68d688b5f79c not found: ID does not exist"
Oct 30 22:02:20.944890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.944851    4282 scope.go:122] "RemoveContainer" containerID="7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0"
Oct 30 22:02:20.953870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.953854    4282 scope.go:122] "RemoveContainer" containerID="7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0"
Oct 30 22:02:20.954177 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:20.954143    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0\": container with ID starting with 7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0 not found: ID does not exist" containerID="7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0"
Oct 30 22:02:20.954282 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.954188    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0"} err="failed to get container status \"7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0\": rpc error: code = NotFound desc = could not find container \"7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0\": container with ID starting with 7b1051684a1cedf7cca6e1d2484f8d89cabf3fdb62fc1b600ac3b29477ad9dd0 not found: ID does not exist"
Oct 30 22:02:20.958149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.958129    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"]
Oct 30 22:02:20.963397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.963370    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-71f39e-c696959b7-6lmmh"]
Oct 30 22:02:20.981972 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.981949    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"]
Oct 30 22:02:20.985147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:20.985127    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-71f39e-67cdf45565-76tk9"]
Oct 30 22:02:21.511493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:21.510841    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174221eb-a62d-49a2-8ffc-e209628e641c" path="/var/lib/kubelet/pods/174221eb-a62d-49a2-8ffc-e209628e641c/volumes"
Oct 30 22:02:21.511493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:21.511421    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2445cdba-b53a-4f2f-898c-7c0b837e8571" path="/var/lib/kubelet/pods/2445cdba-b53a-4f2f-898c-7c0b837e8571/volumes"
Oct 30 22:02:21.908638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:21.908566    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:02:23.049290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:23.049243    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"]
Oct 30 22:02:23.049727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:23.049492    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy" containerID="cri-o://89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23" gracePeriod=30
Oct 30 22:02:23.383552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:23.375406    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"]
Oct 30 22:02:23.383552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:23.375780    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc" podUID="24c40bd3-43dd-4005-9d44-d554f060db78" containerName="maas-api" containerID="cri-o://6a9cea6eb1a4e3c4c3be994e89ecb78445b7f6aed0840dabaf67cf8c19027756" gracePeriod=30
Oct 30 22:02:23.975827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:23.975798    4282 generic.go:356] "Generic (PLEG): container finished" podID="24c40bd3-43dd-4005-9d44-d554f060db78" containerID="6a9cea6eb1a4e3c4c3be994e89ecb78445b7f6aed0840dabaf67cf8c19027756" exitCode=0
Oct 30 22:02:23.975987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:23.975873    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc" event={"ID":"24c40bd3-43dd-4005-9d44-d554f060db78","Type":"ContainerDied","Data":"6a9cea6eb1a4e3c4c3be994e89ecb78445b7f6aed0840dabaf67cf8c19027756"}
Oct 30 22:02:24.011069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.011046    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:02:24.160022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.159350    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c40bd3-43dd-4005-9d44-d554f060db78-kube-api-access-h65nd\" (UniqueName: \"kubernetes.io/projected/24c40bd3-43dd-4005-9d44-d554f060db78-kube-api-access-h65nd\") pod \"24c40bd3-43dd-4005-9d44-d554f060db78\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") "
Oct 30 22:02:24.160022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.159434    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c40bd3-43dd-4005-9d44-d554f060db78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c40bd3-43dd-4005-9d44-d554f060db78-openshift-service-ca\") pod \"24c40bd3-43dd-4005-9d44-d554f060db78\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") "
Oct 30 22:02:24.160022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.159694    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-maas-api-tls\") pod \"24c40bd3-43dd-4005-9d44-d554f060db78\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") "
Oct 30 22:02:24.160022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.159772    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-metrics-certs\") pod \"24c40bd3-43dd-4005-9d44-d554f060db78\" (UID: \"24c40bd3-43dd-4005-9d44-d554f060db78\") "
Oct 30 22:02:24.161792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.161400    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c40bd3-43dd-4005-9d44-d554f060db78-openshift-service-ca" pod "24c40bd3-43dd-4005-9d44-d554f060db78" (UID: "24c40bd3-43dd-4005-9d44-d554f060db78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:24.163690 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.163663    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c40bd3-43dd-4005-9d44-d554f060db78-kube-api-access-h65nd" pod "24c40bd3-43dd-4005-9d44-d554f060db78" (UID: "24c40bd3-43dd-4005-9d44-d554f060db78"). InnerVolumeSpecName "kube-api-access-h65nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:24.164063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.164033    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-metrics-certs" pod "24c40bd3-43dd-4005-9d44-d554f060db78" (UID: "24c40bd3-43dd-4005-9d44-d554f060db78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:24.164148 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.164056    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-maas-api-tls" pod "24c40bd3-43dd-4005-9d44-d554f060db78" (UID: "24c40bd3-43dd-4005-9d44-d554f060db78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:24.261747 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.261715    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.261747 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.261741    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24c40bd3-43dd-4005-9d44-d554f060db78-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.261747 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.261752    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h65nd\" (UniqueName: \"kubernetes.io/projected/24c40bd3-43dd-4005-9d44-d554f060db78-kube-api-access-h65nd\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:24.261977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:24.261762    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24c40bd3-43dd-4005-9d44-d554f060db78-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:25.000406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:25.000356    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc" event={"ID":"24c40bd3-43dd-4005-9d44-d554f060db78","Type":"ContainerDied","Data":"478b3f10bb6c87a0d4a7d6a08e3403db75ee2b7b44e929f07a5020bd14c0b4a6"}
Oct 30 22:02:25.000652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:25.000434    4282 scope.go:122] "RemoveContainer" containerID="6a9cea6eb1a4e3c4c3be994e89ecb78445b7f6aed0840dabaf67cf8c19027756"
Oct 30 22:02:25.000652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:25.000604    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"
Oct 30 22:02:25.043960 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:25.043937    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"]
Oct 30 22:02:25.049022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:25.048999    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71f39e-86697f7c7b-rstbc"]
Oct 30 22:02:25.465307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:25.465280    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c40bd3-43dd-4005-9d44-d554f060db78" path="/var/lib/kubelet/pods/24c40bd3-43dd-4005-9d44-d554f060db78/volumes"
Oct 30 22:02:26.947370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:26.947341    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"]
Oct 30 22:02:26.947934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:26.947570    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy" containerID="cri-o://46064622c183616f56e2444bd35c57343fc92d99dd8fb084d945cd12eb073c35" gracePeriod=30
Oct 30 22:02:28.403729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.403698    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:02:28.462768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.462729    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-istio-token\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-istio-token\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.462928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.462790    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-data\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.462928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.462852    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-kube-api-access-zkrc5\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-kube-api-access-zkrc5\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.463177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.463147    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-data" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:28.463452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.463431    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.465278 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.465251    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-istio-token" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:28.465358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.465285    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-kube-api-access-zkrc5" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "kube-api-access-zkrc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:28.564130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564092    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-certs\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.564317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564185    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16347946-2949-44e3-84fd-3621c321ad08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16347946-2949-44e3-84fd-3621c321ad08-istiod-ca-cert\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.564317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564238    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-socket\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.564317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564282    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-envoy\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.564317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564305    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16347946-2949-44e3-84fd-3621c321ad08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16347946-2949-44e3-84fd-3621c321ad08-istio-podinfo\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.564581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564383    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-credential-socket\") pod \"16347946-2949-44e3-84fd-3621c321ad08\" (UID: \"16347946-2949-44e3-84fd-3621c321ad08\") "
Oct 30 22:02:28.564581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564398    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-certs" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:28.564697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564604    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-socket" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:28.564697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564675    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16347946-2949-44e3-84fd-3621c321ad08-istiod-ca-cert" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:28.564790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.564707    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-credential-socket" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:28.565057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.565033    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.565057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.565056    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.565230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.565066    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.565230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.565079    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrc5\" (UniqueName: \"kubernetes.io/projected/16347946-2949-44e3-84fd-3621c321ad08-kube-api-access-zkrc5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.565230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.565093    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.565230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.565102    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16347946-2949-44e3-84fd-3621c321ad08-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.566636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.566611    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16347946-2949-44e3-84fd-3621c321ad08-istio-podinfo" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:02:28.566636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.566620    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-envoy" pod "16347946-2949-44e3-84fd-3621c321ad08" (UID: "16347946-2949-44e3-84fd-3621c321ad08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:28.666106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.666018    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16347946-2949-44e3-84fd-3621c321ad08-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:28.666106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:28.666045    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16347946-2949-44e3-84fd-3621c321ad08-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:29.038521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.038487    4282 generic.go:356] "Generic (PLEG): container finished" podID="16347946-2949-44e3-84fd-3621c321ad08" containerID="89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23" exitCode=0
Oct 30 22:02:29.038722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.038533    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"
Oct 30 22:02:29.038722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.038567    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" event={"ID":"16347946-2949-44e3-84fd-3621c321ad08","Type":"ContainerDied","Data":"89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23"}
Oct 30 22:02:29.038722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.038620    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh" event={"ID":"16347946-2949-44e3-84fd-3621c321ad08","Type":"ContainerDied","Data":"9a0e9c3e458d488d6670bc4c22fbbf677a2a9b24f18dc35b63a95db4fcbc63ee"}
Oct 30 22:02:29.038722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.038642    4282 scope.go:122] "RemoveContainer" containerID="89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23"
Oct 30 22:02:29.049922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.049904    4282 scope.go:122] "RemoveContainer" containerID="123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d"
Oct 30 22:02:29.060437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.060411    4282 scope.go:122] "RemoveContainer" containerID="89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23"
Oct 30 22:02:29.060766 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:29.060744    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23\": container with ID starting with 89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23 not found: ID does not exist" containerID="89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23"
Oct 30 22:02:29.060870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.060782    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23"} err="failed to get container status \"89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23\": rpc error: code = NotFound desc = could not find container \"89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23\": container with ID starting with 89ec45210fa4ffe0fdf2ee1d04877b8b6fd525747c71f9c274fc432a0ec4ac23 not found: ID does not exist"
Oct 30 22:02:29.060870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.060812    4282 scope.go:122] "RemoveContainer" containerID="123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d"
Oct 30 22:02:29.061058 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:29.061041    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d\": container with ID starting with 123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d not found: ID does not exist" containerID="123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d"
Oct 30 22:02:29.061094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.061065    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d"} err="failed to get container status \"123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d\": rpc error: code = NotFound desc = could not find container \"123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d\": container with ID starting with 123b94b87064dfe2dd359815143852ee1d0870a8cecb3867f6b74a981a15476d not found: ID does not exist"
Oct 30 22:02:29.062017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.061993    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"]
Oct 30 22:02:29.070506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.070455    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3724d9-openshift-default-66f57b958f-nr8qh"]
Oct 30 22:02:29.465967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:29.465936    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16347946-2949-44e3-84fd-3621c321ad08" path="/var/lib/kubelet/pods/16347946-2949-44e3-84fd-3621c321ad08/volumes"
Oct 30 22:02:30.014828 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.014793    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"]
Oct 30 22:02:30.015770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015748    4282 cpu_manager.go:414] "containerMap: removing container" podUID="2445cdba-b53a-4f2f-898c-7c0b837e8571" containerName="payload-pre-processing"
Oct 30 22:02:30.015870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015774    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2445cdba-b53a-4f2f-898c-7c0b837e8571" containerName="payload-pre-processing"
Oct 30 22:02:30.015870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015796    4282 cpu_manager.go:414] "containerMap: removing container" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy"
Oct 30 22:02:30.015870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015805    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy"
Oct 30 22:02:30.015870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015832    4282 cpu_manager.go:414] "containerMap: removing container" podUID="24c40bd3-43dd-4005-9d44-d554f060db78" containerName="maas-api"
Oct 30 22:02:30.015870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015839    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c40bd3-43dd-4005-9d44-d554f060db78" containerName="maas-api"
Oct 30 22:02:30.015870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015852    4282 cpu_manager.go:414] "containerMap: removing container" podUID="174221eb-a62d-49a2-8ffc-e209628e641c" containerName="payload-processing"
Oct 30 22:02:30.015870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015861    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174221eb-a62d-49a2-8ffc-e209628e641c" containerName="payload-processing"
Oct 30 22:02:30.016084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015931    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c0f801ad-9939-400d-b564-c013416f3f6e" containerName="maas-api"
Oct 30 22:02:30.016084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.015939    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f801ad-9939-400d-b564-c013416f3f6e" containerName="maas-api"
Oct 30 22:02:30.016140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.016100    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f801ad-9939-400d-b564-c013416f3f6e" containerName="maas-api"
Oct 30 22:02:30.016140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.016113    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2445cdba-b53a-4f2f-898c-7c0b837e8571" containerName="payload-pre-processing"
Oct 30 22:02:30.016140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.016124    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy"
Oct 30 22:02:30.016140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.016134    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c40bd3-43dd-4005-9d44-d554f060db78" containerName="maas-api"
Oct 30 22:02:30.016257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.016145    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="174221eb-a62d-49a2-8ffc-e209628e641c" containerName="payload-processing"
Oct 30 22:02:30.016257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.016161    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy"
Oct 30 22:02:30.021756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.021736    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.032574 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.032552    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-90b6f5-openshift-default-dockercfg-gq9s5\""
Oct 30 22:02:30.039050 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.039025    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"]
Oct 30 22:02:30.121988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.121955    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-data\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.122159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.121993    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-credential-socket\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.122159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.122019    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-envoy\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.122159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.122111    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-socket\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.122264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.122220    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bb00c7-522d-4e8a-baf0-95897d40452c-istiod-ca-cert\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.122310 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.122297    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6cm2\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-kube-api-access-j6cm2\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.122386 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.122376    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-podinfo\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.122422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.122397    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-certs\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.122458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.122427    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-token\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.231760 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.231718    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6cm2\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-kube-api-access-j6cm2\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.231990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.231818    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-podinfo\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.231990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.231853    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-certs\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.231990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.231898    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-token\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.232174 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.232045    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-data\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.232174 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.232081    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-credential-socket\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.232174 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.232112    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-envoy\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.232174 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.232157    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-socket\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.232389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.232251    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bb00c7-522d-4e8a-baf0-95897d40452c-istiod-ca-cert\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.233105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.233075    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bb00c7-522d-4e8a-baf0-95897d40452c-istiod-ca-cert\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.252495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.251710    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-data\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.252495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.252003    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-certs\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.258325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.253045    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-socket\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.258325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.253363    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-credential-socket\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.271544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.267991    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-envoy\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.290495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.275254    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-podinfo\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.290495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.289112    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6cm2\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-kube-api-access-j6cm2\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.299491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.293034    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-token\") pod \"e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") " pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.347221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.339055    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:30.802921 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.802895    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"]
Oct 30 22:02:30.803394 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:02:30.803374    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bb00c7_522d_4e8a_baf0_95897d40452c.slice/crio-e0f8565e359949ce3e6fc7146f216f0190a8825f6d24247953c78556e4af2865 WatchSource:0}: Error finding container e0f8565e359949ce3e6fc7146f216f0190a8825f6d24247953c78556e4af2865: Status 404 returned error can't find the container with id e0f8565e359949ce3e6fc7146f216f0190a8825f6d24247953c78556e4af2865
Oct 30 22:02:30.806319 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.806280    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:30.806442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.806355    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:30.806442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:30.806392    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:31.058692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:31.058657    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" event={"ID":"b0bb00c7-522d-4e8a-baf0-95897d40452c","Type":"ContainerStarted","Data":"0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d"}
Oct 30 22:02:31.058874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:31.058716    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" event={"ID":"b0bb00c7-522d-4e8a-baf0-95897d40452c","Type":"ContainerStarted","Data":"e0f8565e359949ce3e6fc7146f216f0190a8825f6d24247953c78556e4af2865"}
Oct 30 22:02:31.081171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:31.080556    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podStartSLOduration=2.080537777 podStartE2EDuration="2.080537777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:31.079624362 +0000 UTC m=+1369.737720584" watchObservedRunningTime="2026-10-30 22:02:31.080537777 +0000 UTC m=+1369.738634001"
Oct 30 22:02:31.346308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:31.346273    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:31.348006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:31.347979    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:02:31.348120 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:31.348038    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 22:02:32.068309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.068284    4282 generic.go:356] "Generic (PLEG): container finished" podID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerID="46064622c183616f56e2444bd35c57343fc92d99dd8fb084d945cd12eb073c35" exitCode=0
Oct 30 22:02:32.068737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.068353    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" event={"ID":"81fb080c-08ed-4eee-bcd1-893e4c192f69","Type":"ContainerDied","Data":"46064622c183616f56e2444bd35c57343fc92d99dd8fb084d945cd12eb073c35"}
Oct 30 22:02:32.068737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.068416    4282 scope.go:122] "RemoveContainer" containerID="17c232b350d54e703c535f68a4f0c21f828efc255047d1fd6bb4a9d6db547ee1"
Oct 30 22:02:32.251006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.250984    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 22:02:32.351189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.351161    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:32.414345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414313    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-kube-api-access-qnv9x\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-kube-api-access-qnv9x\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.414607 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414372    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-data\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.414607 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414431    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-envoy\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.414607 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414499    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-certs\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.414774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414647    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-credential-socket\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.414774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414680    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-podinfo\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.414774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414711    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-data" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:32.414774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414752    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fb080c-08ed-4eee-bcd1-893e4c192f69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fb080c-08ed-4eee-bcd1-893e4c192f69-istiod-ca-cert\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.415145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414815    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-socket\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.415145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414856    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-certs" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:32.415145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.414870    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-credential-socket" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:32.415145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.415077    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-token\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-token\") pod \"81fb080c-08ed-4eee-bcd1-893e4c192f69\" (UID: \"81fb080c-08ed-4eee-bcd1-893e4c192f69\") "
Oct 30 22:02:32.415813 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.415152    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-socket" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:32.415813 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.415644    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fb080c-08ed-4eee-bcd1-893e4c192f69-istiod-ca-cert" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:32.416253 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.416228    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.416339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.416260    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.416339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.416276    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.416339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.416291    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fb080c-08ed-4eee-bcd1-893e4c192f69-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.416339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.416305    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.417379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.417331    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-podinfo" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:02:32.417563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.417541    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-envoy" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:32.417691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.417588    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-kube-api-access-qnv9x" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "kube-api-access-qnv9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:32.417889 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.417855    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-token" pod "81fb080c-08ed-4eee-bcd1-893e4c192f69" (UID: "81fb080c-08ed-4eee-bcd1-893e4c192f69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:32.552307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.546191    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.552307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.546228    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnv9x\" (UniqueName: \"kubernetes.io/projected/81fb080c-08ed-4eee-bcd1-893e4c192f69-kube-api-access-qnv9x\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.552307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.546245    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.552307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.546259    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fb080c-08ed-4eee-bcd1-893e4c192f69-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:32.577984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.570552    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"]
Oct 30 22:02:32.577984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:32.570819    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy" containerID="cri-o://47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228" gracePeriod=30
Oct 30 22:02:33.090702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:33.090524    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn" event={"ID":"81fb080c-08ed-4eee-bcd1-893e4c192f69","Type":"ContainerDied","Data":"28fac0ed86de509c27306b4f9bf1bcfe7c7240bc940502b5ea095137f3e7d257"}
Oct 30 22:02:33.090702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:33.090594    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:33.090702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:33.090625    4282 scope.go:122] "RemoveContainer" containerID="46064622c183616f56e2444bd35c57343fc92d99dd8fb084d945cd12eb073c35"
Oct 30 22:02:33.094051 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:33.094026    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:02:33.095691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:33.095656    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"
Oct 30 22:02:33.148787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:33.148749    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"]
Oct 30 22:02:33.161445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:33.160226    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-jc2tn"]
Oct 30 22:02:33.485362 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:33.485333    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" path="/var/lib/kubelet/pods/81fb080c-08ed-4eee-bcd1-893e4c192f69/volumes"
Oct 30 22:02:34.738202 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:34.738169    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:02:35.047654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.047576    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"]
Oct 30 22:02:35.049089 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.049058    4282 cpu_manager.go:414] "containerMap: removing container" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy"
Oct 30 22:02:35.049089 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.049088    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy"
Oct 30 22:02:35.049318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.049114    4282 cpu_manager.go:414] "containerMap: removing container" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy"
Oct 30 22:02:35.049318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.049121    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy"
Oct 30 22:02:35.049318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.049190    4282 cpu_manager.go:414] "containerMap: removing container" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy"
Oct 30 22:02:35.049318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.049199    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16347946-2949-44e3-84fd-3621c321ad08" containerName="istio-proxy"
Oct 30 22:02:35.049598 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.049579    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy"
Oct 30 22:02:35.049660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.049604    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fb080c-08ed-4eee-bcd1-893e4c192f69" containerName="istio-proxy"
Oct 30 22:02:35.083503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.074892    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:35.085674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.085623    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-90b6f5-dockercfg-swjmv\""
Oct 30 22:02:35.086292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.086259    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-90b6f5\""
Oct 30 22:02:35.100200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.100171    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"]
Oct 30 22:02:35.115034 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.115010    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"]
Oct 30 22:02:35.121199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.121177    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:35.250496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.242575    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48\" (UID: \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:35.250496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.242732    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrjh\" (UniqueName: \"kubernetes.io/projected/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-kube-api-access-ztrjh\") pod \"payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48\" (UID: \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:35.250496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.242854    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vwz\" (UniqueName: \"kubernetes.io/projected/8d984bb8-3ed8-4790-827a-4292af0cfcd2-kube-api-access-74vwz\") pod \"payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc\" (UID: \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:35.250496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.242960    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d984bb8-3ed8-4790-827a-4292af0cfcd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc\" (UID: \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:35.250496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.244287    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"]
Oct 30 22:02:35.460617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.348677    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrjh\" (UniqueName: \"kubernetes.io/projected/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-kube-api-access-ztrjh\") pod \"payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48\" (UID: \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:35.460617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.348822    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vwz\" (UniqueName: \"kubernetes.io/projected/8d984bb8-3ed8-4790-827a-4292af0cfcd2-kube-api-access-74vwz\") pod \"payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc\" (UID: \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:35.460617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.348901    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d984bb8-3ed8-4790-827a-4292af0cfcd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc\" (UID: \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:35.460617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.388737    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48\" (UID: \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:35.460617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.408011    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d984bb8-3ed8-4790-827a-4292af0cfcd2-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc\" (UID: \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:35.460617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.418022    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48\" (UID: \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:35.520644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.519392    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrjh\" (UniqueName: \"kubernetes.io/projected/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-kube-api-access-ztrjh\") pod \"payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48\" (UID: \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:35.523028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.523002    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vwz\" (UniqueName: \"kubernetes.io/projected/8d984bb8-3ed8-4790-827a-4292af0cfcd2-kube-api-access-74vwz\") pod \"payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc\" (UID: \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:35.775095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.755153    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:35.817209 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:35.817176    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:36.131437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:36.131407    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"]
Oct 30 22:02:36.173495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:36.152520    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"]
Oct 30 22:02:37.167223 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.167181    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc" event={"ID":"8d984bb8-3ed8-4790-827a-4292af0cfcd2","Type":"ContainerStarted","Data":"11f7ae5ed6c8d52371cd85c53a71551950d4d8bd798268ec298304281a07db11"}
Oct 30 22:02:37.178617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.178578    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc" event={"ID":"8d984bb8-3ed8-4790-827a-4292af0cfcd2","Type":"ContainerStarted","Data":"25f5d03e40d815d1cd3a3d1fff0ab7ba7c66760346549afd4320f5138fedc651"}
Oct 30 22:02:37.178877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.178859    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:37.196102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.196065    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48" event={"ID":"501a2c31-fb1e-47b9-acf7-aff6367ef3ba","Type":"ContainerStarted","Data":"ae96f8ae3b174af0137035b97c659a45c4bbbbdf71d538fbef3df5064ac6d0ad"}
Oct 30 22:02:37.196382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.196357    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48" event={"ID":"501a2c31-fb1e-47b9-acf7-aff6367ef3ba","Type":"ContainerStarted","Data":"c5a9a1c558a34c142392ef9927b377ff058b2f9b77ab06341b5eee140e06a1d7"}
Oct 30 22:02:37.208501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.198531    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:37.236507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.236423    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc" podStartSLOduration=2.236401851 podStartE2EDuration="2.236401851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:37.194337592 +0000 UTC m=+1375.852433820" watchObservedRunningTime="2026-10-30 22:02:37.236401851 +0000 UTC m=+1375.894498077"
Oct 30 22:02:37.248801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.248746    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48" podStartSLOduration=2.248729348 podStartE2EDuration="2.248729348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:37.234543254 +0000 UTC m=+1375.892639480" watchObservedRunningTime="2026-10-30 22:02:37.248729348 +0000 UTC m=+1375.906825572"
Oct 30 22:02:37.921806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.921776    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.987401    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-kube-api-access-wdf8c\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-kube-api-access-wdf8c\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.987504    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-certs\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.987570    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-socket\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.987641    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-podinfo\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.987814    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istiod-ca-cert\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.987863    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-credential-socket\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.987897    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-token\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.987950    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-envoy\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.991491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.988004    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-data\") pod \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\" (UID: \"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d\") "
Oct 30 22:02:37.996503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.992192    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-data" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.000500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:37.999688    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istiod-ca-cert" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:38.013638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.011414    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-socket" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.013638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.012579    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-credential-socket" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.031501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.022580    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-certs" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.098229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.091090    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.098229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.091132    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.098229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.091147    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.098229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.091161    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.098229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.091175    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.112921 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.112837    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-podinfo" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:02:38.126303 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.118661    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-kube-api-access-wdf8c" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "kube-api-access-wdf8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:38.128151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.127975    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-envoy" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:02:38.130960 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.130927    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-token" pod "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" (UID: "6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:38.193094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.193063    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.193094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.193096    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.193621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.193109    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.193621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.193122    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdf8c\" (UniqueName: \"kubernetes.io/projected/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d-kube-api-access-wdf8c\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:38.206194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.206163    4282 generic.go:356] "Generic (PLEG): container finished" podID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerID="47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228" exitCode=0
Oct 30 22:02:38.206339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.206217    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"
Oct 30 22:02:38.206339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.206288    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" event={"ID":"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d","Type":"ContainerDied","Data":"47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228"}
Oct 30 22:02:38.206489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.206346    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g" event={"ID":"6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d","Type":"ContainerDied","Data":"8a8e41df50f9a161319c593f2c9533dbd74b793749bc310bd211082b2260de9b"}
Oct 30 22:02:38.206489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.206377    4282 scope.go:122] "RemoveContainer" containerID="47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228"
Oct 30 22:02:38.218124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.218106    4282 scope.go:122] "RemoveContainer" containerID="8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2"
Oct 30 22:02:38.227682 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.227657    4282 scope.go:122] "RemoveContainer" containerID="47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228"
Oct 30 22:02:38.227978 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:38.227956    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228\": container with ID starting with 47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228 not found: ID does not exist" containerID="47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228"
Oct 30 22:02:38.228062 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.227992    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228"} err="failed to get container status \"47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228\": rpc error: code = NotFound desc = could not find container \"47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228\": container with ID starting with 47f8b46a7e51710f4beb076d6991817997c2d779543c89b4de5dbc9e8eb3c228 not found: ID does not exist"
Oct 30 22:02:38.228062 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.228018    4282 scope.go:122] "RemoveContainer" containerID="8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2"
Oct 30 22:02:38.228329 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:02:38.228309    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2\": container with ID starting with 8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2 not found: ID does not exist" containerID="8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2"
Oct 30 22:02:38.228377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.228338    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2"} err="failed to get container status \"8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2\": rpc error: code = NotFound desc = could not find container \"8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2\": container with ID starting with 8b9f18bb7d19d1fc3f6ed296b1f79329d33e1f91a1f7487d818aa2ef74b413d2 not found: ID does not exist"
Oct 30 22:02:38.230525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.230505    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"]
Oct 30 22:02:38.236632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:38.236612    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-lnk5g"]
Oct 30 22:02:39.492543 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:39.490985    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" path="/var/lib/kubelet/pods/6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d/volumes"
Oct 30 22:02:41.424395 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.424355    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"]
Oct 30 22:02:41.426141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.426115    4282 cpu_manager.go:414] "containerMap: removing container" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy"
Oct 30 22:02:41.427639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.427217    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy"
Oct 30 22:02:41.427639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.427266    4282 cpu_manager.go:414] "containerMap: removing container" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy"
Oct 30 22:02:41.427639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.427275    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy"
Oct 30 22:02:41.427639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.427504    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy"
Oct 30 22:02:41.427639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.427523    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2f09f7-ffd1-4f55-82a6-5c7b8f90c78d" containerName="istio-proxy"
Oct 30 22:02:41.437348 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.434091    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.445352 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.445150    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-90b6f5-serving-cert\""
Oct 30 22:02:41.448925 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.448461    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"]
Oct 30 22:02:41.450400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.450108    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.450400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.450177    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0905a38-269b-4f46-be5b-aca1ed6f954c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.450400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.450326    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-metrics-certs\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.450400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.450359    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7vjv\" (UniqueName: \"kubernetes.io/projected/b0905a38-269b-4f46-be5b-aca1ed6f954c-kube-api-access-q7vjv\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.551912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.551884    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.552109 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.551945    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0905a38-269b-4f46-be5b-aca1ed6f954c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.552109 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.552049    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-metrics-certs\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.552109 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.552077    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7vjv\" (UniqueName: \"kubernetes.io/projected/b0905a38-269b-4f46-be5b-aca1ed6f954c-kube-api-access-q7vjv\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.553063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.553031    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0905a38-269b-4f46-be5b-aca1ed6f954c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.555178 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.555154    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-metrics-certs\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.555263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.555192    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.567014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.566980    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7vjv\" (UniqueName: \"kubernetes.io/projected/b0905a38-269b-4f46-be5b-aca1ed6f954c-kube-api-access-q7vjv\") pod \"maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:41.774221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:41.774141    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:42.138576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:42.138372    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"]
Oct 30 22:02:42.186064 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:02:42.186023    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0905a38_269b_4f46_be5b_aca1ed6f954c.slice/crio-d206659394605115af00cc96ccddc576eb18ca5a6c9557a85ca69c4d311f4213 WatchSource:0}: Error finding container d206659394605115af00cc96ccddc576eb18ca5a6c9557a85ca69c4d311f4213: Status 404 returned error can't find the container with id d206659394605115af00cc96ccddc576eb18ca5a6c9557a85ca69c4d311f4213
Oct 30 22:02:42.247830 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:42.247799    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs" event={"ID":"b0905a38-269b-4f46-be5b-aca1ed6f954c","Type":"ContainerStarted","Data":"d206659394605115af00cc96ccddc576eb18ca5a6c9557a85ca69c4d311f4213"}
Oct 30 22:02:42.251584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:42.251558    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:42.251740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:42.251605    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:42.982280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:42.982192    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"]
Oct 30 22:02:42.982729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:42.982445    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4" podUID="63444212-7be1-4758-b943-9d5be34f4d1e" containerName="maas-api" containerID="cri-o://ec90c3f39f1df786b0a9d6ca7d86f25c59772a30ccce4f7451ea0beb32963398" gracePeriod=30
Oct 30 22:02:43.466627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.466589    4282 generic.go:356] "Generic (PLEG): container finished" podID="63444212-7be1-4758-b943-9d5be34f4d1e" containerID="ec90c3f39f1df786b0a9d6ca7d86f25c59772a30ccce4f7451ea0beb32963398" exitCode=0
Oct 30 22:02:43.470577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.470520    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs" event={"ID":"b0905a38-269b-4f46-be5b-aca1ed6f954c","Type":"ContainerStarted","Data":"d6e60be2c88bc0bdbb24caca11369ad30d4223cf2e307e96006e7f8fd85194da"}
Oct 30 22:02:43.470761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.470583    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:43.470761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.470602    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4" event={"ID":"63444212-7be1-4758-b943-9d5be34f4d1e","Type":"ContainerDied","Data":"ec90c3f39f1df786b0a9d6ca7d86f25c59772a30ccce4f7451ea0beb32963398"}
Oct 30 22:02:43.493457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.493407    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs" podStartSLOduration=2.019388445 podStartE2EDuration="2.493393165s" totalImagesPullingTime="474.00472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:43.487856191 +0000 UTC m=+1382.145952418" watchObservedRunningTime="2026-10-30 22:02:43.493393165 +0000 UTC m=+1382.151489462"
Oct 30 22:02:43.503694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.503670    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:02:43.584392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.584362    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63444212-7be1-4758-b943-9d5be34f4d1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63444212-7be1-4758-b943-9d5be34f4d1e-openshift-service-ca\") pod \"63444212-7be1-4758-b943-9d5be34f4d1e\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") "
Oct 30 22:02:43.584982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.584955    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-maas-api-tls\") pod \"63444212-7be1-4758-b943-9d5be34f4d1e\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") "
Oct 30 22:02:43.585153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.585024    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63444212-7be1-4758-b943-9d5be34f4d1e-kube-api-access-jvlzf\" (UniqueName: \"kubernetes.io/projected/63444212-7be1-4758-b943-9d5be34f4d1e-kube-api-access-jvlzf\") pod \"63444212-7be1-4758-b943-9d5be34f4d1e\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") "
Oct 30 22:02:43.585153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.585109    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-metrics-certs\") pod \"63444212-7be1-4758-b943-9d5be34f4d1e\" (UID: \"63444212-7be1-4758-b943-9d5be34f4d1e\") "
Oct 30 22:02:43.585673 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.585644    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63444212-7be1-4758-b943-9d5be34f4d1e-openshift-service-ca" pod "63444212-7be1-4758-b943-9d5be34f4d1e" (UID: "63444212-7be1-4758-b943-9d5be34f4d1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:43.585970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.585953    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63444212-7be1-4758-b943-9d5be34f4d1e-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:43.590855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.590833    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-maas-api-tls" pod "63444212-7be1-4758-b943-9d5be34f4d1e" (UID: "63444212-7be1-4758-b943-9d5be34f4d1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:43.592140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.592114    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-metrics-certs" pod "63444212-7be1-4758-b943-9d5be34f4d1e" (UID: "63444212-7be1-4758-b943-9d5be34f4d1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:43.592239 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.592190    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63444212-7be1-4758-b943-9d5be34f4d1e-kube-api-access-jvlzf" pod "63444212-7be1-4758-b943-9d5be34f4d1e" (UID: "63444212-7be1-4758-b943-9d5be34f4d1e"). InnerVolumeSpecName "kube-api-access-jvlzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:43.687741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.687709    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:43.687741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.687742    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvlzf\" (UniqueName: \"kubernetes.io/projected/63444212-7be1-4758-b943-9d5be34f4d1e-kube-api-access-jvlzf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:43.687953 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.687756    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63444212-7be1-4758-b943-9d5be34f4d1e-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:43.918571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:43.917425    4282 scope.go:122] "RemoveContainer" containerID="f31f67b42b5e2659f32439918c3b0372d6559ae3751f6c992396fa1984774b0c"
Oct 30 22:02:44.528912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.519819    4282 generic.go:356] "Generic (PLEG): container finished" podID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerID="56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81" exitCode=0
Oct 30 22:02:44.528912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.519885    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" event={"ID":"9eda7d94-5787-4a6d-97ce-bd2337998a88","Type":"ContainerDied","Data":"56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81"}
Oct 30 22:02:44.528912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.520738    4282 scope.go:122] "RemoveContainer" containerID="56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81"
Oct 30 22:02:44.528912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.523273    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:44.528912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.523367    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:44.528912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.523411    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:02:44.536630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.536607    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"
Oct 30 22:02:44.537026 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.536921    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4" event={"ID":"63444212-7be1-4758-b943-9d5be34f4d1e","Type":"ContainerDied","Data":"dabcbf5828ed485a8ddeecdfc6814a50da4f3023ec48b886cbee102db6c2af0b"}
Oct 30 22:02:44.537129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.537071    4282 scope.go:122] "RemoveContainer" containerID="ec90c3f39f1df786b0a9d6ca7d86f25c59772a30ccce4f7451ea0beb32963398"
Oct 30 22:02:44.592352 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.592299    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"]
Oct 30 22:02:44.597056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:44.597031    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c097ae-c79d7854d-lrqw4"]
Oct 30 22:02:45.464800 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:45.464771    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63444212-7be1-4758-b943-9d5be34f4d1e" path="/var/lib/kubelet/pods/63444212-7be1-4758-b943-9d5be34f4d1e/volumes"
Oct 30 22:02:45.545417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:45.545377    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" event={"ID":"9eda7d94-5787-4a6d-97ce-bd2337998a88","Type":"ContainerStarted","Data":"8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0"}
Oct 30 22:02:46.078114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:46.078022    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:02:46.078357 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:46.078152    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:02:46.078357 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:46.078195    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:02:47.117491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:47.115724    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:02:47.117491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:47.115804    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:02:48.077891 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:48.077856    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:02:48.078066 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:48.077924    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:02:48.209146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:48.209112    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:02:48.403640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:48.403565    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"]
Oct 30 22:02:48.404072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:48.404039    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj" podUID="00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" containerName="maas-api" containerID="cri-o://3c34094ffea8a2e8a9e4edcedddd41e8c8f73a7dad4e3cf36f4a620ce0f8897a" gracePeriod=30
Oct 30 22:02:48.637499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:48.622312    4282 generic.go:356] "Generic (PLEG): container finished" podID="00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" containerID="3c34094ffea8a2e8a9e4edcedddd41e8c8f73a7dad4e3cf36f4a620ce0f8897a" exitCode=0
Oct 30 22:02:48.637499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:48.622501    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj" event={"ID":"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc","Type":"ContainerDied","Data":"3c34094ffea8a2e8a9e4edcedddd41e8c8f73a7dad4e3cf36f4a620ce0f8897a"}
Oct 30 22:02:49.122561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.114625    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:02:49.122561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.114701    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:02:49.173499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.172876    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:02:49.220541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.220510    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:02:49.280501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.280120    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-kube-api-access-d6z5r\" (UniqueName: \"kubernetes.io/projected/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-kube-api-access-d6z5r\") pod \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") "
Oct 30 22:02:49.280501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.280257    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-openshift-service-ca\") pod \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") "
Oct 30 22:02:49.280501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.280368    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-metrics-certs\") pod \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") "
Oct 30 22:02:49.280799 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.280527    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-maas-api-tls\") pod \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\" (UID: \"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc\") "
Oct 30 22:02:49.286521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.285075    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-openshift-service-ca" pod "00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" (UID: "00009cbb-b4c6-4cb8-a3b8-a87e56a255bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:49.286878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.286848    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-metrics-certs" pod "00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" (UID: "00009cbb-b4c6-4cb8-a3b8-a87e56a255bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:49.287398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.287377    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-maas-api-tls" pod "00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" (UID: "00009cbb-b4c6-4cb8-a3b8-a87e56a255bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:49.289614 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.289591    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-kube-api-access-d6z5r" pod "00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" (UID: "00009cbb-b4c6-4cb8-a3b8-a87e56a255bc"). InnerVolumeSpecName "kube-api-access-d6z5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:49.390213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.389727    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:49.390213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.389763    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:49.390213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.389780    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:49.390213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.389794    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6z5r\" (UniqueName: \"kubernetes.io/projected/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc-kube-api-access-d6z5r\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:49.543988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.543958    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:02:49.630970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.630940    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"
Oct 30 22:02:49.631139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.630986    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj" event={"ID":"00009cbb-b4c6-4cb8-a3b8-a87e56a255bc","Type":"ContainerDied","Data":"7d148386458ef62a42945daa8b8e9018f7a95996d652dc2f8e0c63f8d0e1eeaf"}
Oct 30 22:02:49.631139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.631043    4282 scope.go:122] "RemoveContainer" containerID="3c34094ffea8a2e8a9e4edcedddd41e8c8f73a7dad4e3cf36f4a620ce0f8897a"
Oct 30 22:02:49.681531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.681461    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"]
Oct 30 22:02:49.686742 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:49.686717    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a1ead7-bdddbfc7f-w22rj"]
Oct 30 22:02:50.077890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:50.077852    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:02:50.078115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:50.077906    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:02:50.407440 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:50.407341    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:02:51.076847 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:51.076806    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Oct 30 22:02:51.077019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:51.076888    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Oct 30 22:02:51.491130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:51.491088    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" path="/var/lib/kubelet/pods/00009cbb-b4c6-4cb8-a3b8-a87e56a255bc/volumes"
Oct 30 22:02:52.146290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:52.146254    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:02:52.148924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:52.148791    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:02:52.159684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:52.159663    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:02:55.174933 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:55.174845    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"]
Oct 30 22:02:55.175412 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:55.175116    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" containerID="cri-o://e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc" gracePeriod=30
Oct 30 22:02:57.242402 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:57.242366    4282 patch_prober.go:29] interesting pod/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:02:57.242835 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:02:57.242432    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:03:00.546202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.546179    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 22:03:00.647144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647111    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-token\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-token\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647152    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-socket\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647228    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-podinfo\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647330    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-credential-socket\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647360    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-data\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647379    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-kube-api-access-87lk9\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-kube-api-access-87lk9\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647396    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-envoy\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647430    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istiod-ca-cert\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647515    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-certs\") pod \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\" (UID: \"aa4a5ed4-af4c-4c26-8600-bae241eb1982\") "
Oct 30 22:03:00.647761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647514    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-socket" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:00.647761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647660    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-data" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:00.647928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647861    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-credential-socket" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:00.648062 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.648037    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-certs" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:00.648153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.647970    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istiod-ca-cert" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:00.648416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.648396    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.648526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.648423    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.648526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.648438    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.648526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.648452    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.648526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.648496    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.649689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.649670    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-envoy" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:00.649902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.649881    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-kube-api-access-87lk9" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "kube-api-access-87lk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:00.650064 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.650041    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-token" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:00.650157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.650067    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-podinfo" pod "aa4a5ed4-af4c-4c26-8600-bae241eb1982" (UID: "aa4a5ed4-af4c-4c26-8600-bae241eb1982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:00.750415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.750381    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87lk9\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-kube-api-access-87lk9\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.750415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.750410    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.750415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.750420    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:00.750686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:00.750428    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa4a5ed4-af4c-4c26-8600-bae241eb1982-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:01.018797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.018762    4282 generic.go:356] "Generic (PLEG): container finished" podID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerID="e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc" exitCode=0
Oct 30 22:03:01.018986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.018826    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"
Oct 30 22:03:01.018986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.018837    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" event={"ID":"aa4a5ed4-af4c-4c26-8600-bae241eb1982","Type":"ContainerDied","Data":"e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc"}
Oct 30 22:03:01.018986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.018912    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2" event={"ID":"aa4a5ed4-af4c-4c26-8600-bae241eb1982","Type":"ContainerDied","Data":"b0576ec7e956eecbd54d1c102e2a0cdbc86ffdf0ce579ba3e72ea58e6d925c65"}
Oct 30 22:03:01.018986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.018933    4282 scope.go:122] "RemoveContainer" containerID="e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc"
Oct 30 22:03:01.029879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.029862    4282 scope.go:122] "RemoveContainer" containerID="e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548"
Oct 30 22:03:01.038872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.038855    4282 scope.go:122] "RemoveContainer" containerID="e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc"
Oct 30 22:03:01.039102 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:01.039082    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc\": container with ID starting with e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc not found: ID does not exist" containerID="e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc"
Oct 30 22:03:01.039153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.039111    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc"} err="failed to get container status \"e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc\": rpc error: code = NotFound desc = could not find container \"e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc\": container with ID starting with e9c3ef28a40b8c8bdf9757c99b967b9493d36b024fe4209c3e93155380e896dc not found: ID does not exist"
Oct 30 22:03:01.039153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.039127    4282 scope.go:122] "RemoveContainer" containerID="e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548"
Oct 30 22:03:01.039367 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:01.039350    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548\": container with ID starting with e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548 not found: ID does not exist" containerID="e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548"
Oct 30 22:03:01.039414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.039372    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548"} err="failed to get container status \"e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548\": rpc error: code = NotFound desc = could not find container \"e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548\": container with ID starting with e5a9878ab74c9af3f8fc4fb7da246629208bc6ec0429222cde54c2bcedbfc548 not found: ID does not exist"
Oct 30 22:03:01.043807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.043786    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"]
Oct 30 22:03:01.049348 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.049329    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3e3670-openshift-default-74f5687cd5-dscf2"]
Oct 30 22:03:01.464961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:01.464929    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" path="/var/lib/kubelet/pods/aa4a5ed4-af4c-4c26-8600-bae241eb1982/volumes"
Oct 30 22:03:02.754165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754134    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"]
Oct 30 22:03:02.754734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754721    4282 cpu_manager.go:414] "containerMap: removing container" podUID="00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" containerName="maas-api"
Oct 30 22:03:02.754790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754735    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" containerName="maas-api"
Oct 30 22:03:02.754790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754767    4282 cpu_manager.go:414] "containerMap: removing container" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy"
Oct 30 22:03:02.754790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754771    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy"
Oct 30 22:03:02.754790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754782    4282 cpu_manager.go:414] "containerMap: removing container" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy"
Oct 30 22:03:02.754790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754787    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy"
Oct 30 22:03:02.754946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754795    4282 cpu_manager.go:414] "containerMap: removing container" podUID="63444212-7be1-4758-b943-9d5be34f4d1e" containerName="maas-api"
Oct 30 22:03:02.754946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754799    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63444212-7be1-4758-b943-9d5be34f4d1e" containerName="maas-api"
Oct 30 22:03:02.754946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754887    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy"
Oct 30 22:03:02.754946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754896    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4a5ed4-af4c-4c26-8600-bae241eb1982" containerName="istio-proxy"
Oct 30 22:03:02.754946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754904    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="63444212-7be1-4758-b943-9d5be34f4d1e" containerName="maas-api"
Oct 30 22:03:02.754946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.754913    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="00009cbb-b4c6-4cb8-a3b8-a87e56a255bc" containerName="maas-api"
Oct 30 22:03:02.759648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.759631    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:02.767362 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.767056    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5d81a2\""
Oct 30 22:03:02.772115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.772089    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"]
Oct 30 22:03:02.836854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.836828    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"]
Oct 30 22:03:02.840928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.840908    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:02.848277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.848253    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5d81a2-dockercfg-25f7b\""
Oct 30 22:03:02.856278 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.856258    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"]
Oct 30 22:03:02.891759 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.891707    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b8e600-068c-413b-b73c-244dd27e942c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq\" (UID: \"38b8e600-068c-413b-b73c-244dd27e942c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:02.891899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.891882    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktrbg\" (UniqueName: \"kubernetes.io/projected/38b8e600-068c-413b-b73c-244dd27e942c-kube-api-access-ktrbg\") pod \"payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq\" (UID: \"38b8e600-068c-413b-b73c-244dd27e942c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:02.891957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.891914    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxcv\" (UniqueName: \"kubernetes.io/projected/5dc521f2-3dfa-470a-b780-ee43a2a113c4-kube-api-access-kqxcv\") pod \"payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz\" (UID: \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:02.892104 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.892076    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc521f2-3dfa-470a-b780-ee43a2a113c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz\" (UID: \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:02.998495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.997408    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc521f2-3dfa-470a-b780-ee43a2a113c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz\" (UID: \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:02.998495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.997536    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b8e600-068c-413b-b73c-244dd27e942c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq\" (UID: \"38b8e600-068c-413b-b73c-244dd27e942c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:02.998495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.997744    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktrbg\" (UniqueName: \"kubernetes.io/projected/38b8e600-068c-413b-b73c-244dd27e942c-kube-api-access-ktrbg\") pod \"payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq\" (UID: \"38b8e600-068c-413b-b73c-244dd27e942c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:02.998495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:02.997788    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxcv\" (UniqueName: \"kubernetes.io/projected/5dc521f2-3dfa-470a-b780-ee43a2a113c4-kube-api-access-kqxcv\") pod \"payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz\" (UID: \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:03.024530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:03.024427    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxcv\" (UniqueName: \"kubernetes.io/projected/5dc521f2-3dfa-470a-b780-ee43a2a113c4-kube-api-access-kqxcv\") pod \"payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz\" (UID: \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:03.025572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:03.025544    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b8e600-068c-413b-b73c-244dd27e942c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq\" (UID: \"38b8e600-068c-413b-b73c-244dd27e942c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:03.044620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:03.044588    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktrbg\" (UniqueName: \"kubernetes.io/projected/38b8e600-068c-413b-b73c-244dd27e942c-kube-api-access-ktrbg\") pod \"payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq\" (UID: \"38b8e600-068c-413b-b73c-244dd27e942c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:03.068496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:03.066037    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc521f2-3dfa-470a-b780-ee43a2a113c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz\" (UID: \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:03.093491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:03.092486    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:03.168414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:03.163996    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:03.487659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:03.487629    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"]
Oct 30 22:03:03.488337 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:03:03.488312    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc521f2_3dfa_470a_b780_ee43a2a113c4.slice/crio-3ac5ab959ee45480863d7c111685714acf859e1ed1cda38cf7418b3ae8415abf WatchSource:0}: Error finding container 3ac5ab959ee45480863d7c111685714acf859e1ed1cda38cf7418b3ae8415abf: Status 404 returned error can't find the container with id 3ac5ab959ee45480863d7c111685714acf859e1ed1cda38cf7418b3ae8415abf
Oct 30 22:03:03.519571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:03.519545    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"]
Oct 30 22:03:03.521355 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:03:03.521329    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b8e600_068c_413b_b73c_244dd27e942c.slice/crio-21b3ca17c6eb9d29adc707059484fb930e2b512ae567fd8e3d7c8d21c85179f3 WatchSource:0}: Error finding container 21b3ca17c6eb9d29adc707059484fb930e2b512ae567fd8e3d7c8d21c85179f3: Status 404 returned error can't find the container with id 21b3ca17c6eb9d29adc707059484fb930e2b512ae567fd8e3d7c8d21c85179f3
Oct 30 22:03:04.126491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:04.116367    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz" event={"ID":"5dc521f2-3dfa-470a-b780-ee43a2a113c4","Type":"ContainerStarted","Data":"d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60"}
Oct 30 22:03:04.126491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:04.116435    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz" event={"ID":"5dc521f2-3dfa-470a-b780-ee43a2a113c4","Type":"ContainerStarted","Data":"3ac5ab959ee45480863d7c111685714acf859e1ed1cda38cf7418b3ae8415abf"}
Oct 30 22:03:04.126491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:04.117865    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:04.126491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:04.121344    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq" event={"ID":"38b8e600-068c-413b-b73c-244dd27e942c","Type":"ContainerStarted","Data":"eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28"}
Oct 30 22:03:04.126491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:04.121389    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq" event={"ID":"38b8e600-068c-413b-b73c-244dd27e942c","Type":"ContainerStarted","Data":"21b3ca17c6eb9d29adc707059484fb930e2b512ae567fd8e3d7c8d21c85179f3"}
Oct 30 22:03:04.126491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:04.122266    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:04.174655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:04.174413    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq" podStartSLOduration=2.174391019 podStartE2EDuration="2.174391019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:04.172080364 +0000 UTC m=+1402.830176587" watchObservedRunningTime="2026-10-30 22:03:04.174391019 +0000 UTC m=+1402.832487244"
Oct 30 22:03:04.177937 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:04.177895    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz" podStartSLOduration=2.177882211 podStartE2EDuration="2.177882211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:04.144408114 +0000 UTC m=+1402.802504338" watchObservedRunningTime="2026-10-30 22:03:04.177882211 +0000 UTC m=+1402.835978437"
Oct 30 22:03:04.649615 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:04.649582    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:03:05.208773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:05.208741    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"]
Oct 30 22:03:05.228367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:05.228337    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"]
Oct 30 22:03:07.727891 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.727861    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"]
Oct 30 22:03:07.732591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.732565    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.739580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.739549    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5d81a2-serving-cert\""
Oct 30 22:03:07.743542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.743516    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"]
Oct 30 22:03:07.869523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.869488    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.869714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.869653    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49d2g\" (UniqueName: \"kubernetes.io/projected/5f795a2a-499e-44cf-83c3-f2c34e94df56-kube-api-access-49d2g\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.869808 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.869784    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f795a2a-499e-44cf-83c3-f2c34e94df56-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.869868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.869840    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.890850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.890819    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"]
Oct 30 22:03:07.904766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.904737    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"]
Oct 30 22:03:07.971962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.971931    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f795a2a-499e-44cf-83c3-f2c34e94df56-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.972128 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.971996    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.972128 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.972128    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.972263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.972204    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49d2g\" (UniqueName: \"kubernetes.io/projected/5f795a2a-499e-44cf-83c3-f2c34e94df56-kube-api-access-49d2g\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.972681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.972656    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f795a2a-499e-44cf-83c3-f2c34e94df56-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.974786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.974760    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.974885 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.974794    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:07.984349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:07.984300    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49d2g\" (UniqueName: \"kubernetes.io/projected/5f795a2a-499e-44cf-83c3-f2c34e94df56-kube-api-access-49d2g\") pod \"maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:08.048519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:08.048451    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:08.597194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:08.586810    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"]
Oct 30 22:03:09.160086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:09.160053    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h" event={"ID":"5f795a2a-499e-44cf-83c3-f2c34e94df56","Type":"ContainerStarted","Data":"e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa"}
Oct 30 22:03:09.160453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:09.160116    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h" event={"ID":"5f795a2a-499e-44cf-83c3-f2c34e94df56","Type":"ContainerStarted","Data":"65336f1661cbb57d0473dfd5993ce25be97d8add78c18852a0a14c67443e6d5a"}
Oct 30 22:03:09.160453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:09.160142    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:09.182320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:09.182270    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h" podStartSLOduration=1.731131083 podStartE2EDuration="2.182254729s" totalImagesPullingTime="451.123646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:09.17814882 +0000 UTC m=+1407.836245047" watchObservedRunningTime="2026-10-30 22:03:09.182254729 +0000 UTC m=+1407.840350951"
Oct 30 22:03:09.922919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:09.922882    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"]
Oct 30 22:03:09.923246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:09.923162    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2" podUID="ea05b385-c1b2-4020-839b-843de45e95bf" containerName="maas-api" containerID="cri-o://b0744e3222314b4c3eb252c51c674da3fa670a811facaf762b003dee9d5c9a65" gracePeriod=30
Oct 30 22:03:10.170580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.170546    4282 generic.go:356] "Generic (PLEG): container finished" podID="ea05b385-c1b2-4020-839b-843de45e95bf" containerID="b0744e3222314b4c3eb252c51c674da3fa670a811facaf762b003dee9d5c9a65" exitCode=0
Oct 30 22:03:10.170982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.170617    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2" event={"ID":"ea05b385-c1b2-4020-839b-843de45e95bf","Type":"ContainerDied","Data":"b0744e3222314b4c3eb252c51c674da3fa670a811facaf762b003dee9d5c9a65"}
Oct 30 22:03:10.209744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.209718    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5" podUID="a8e94194-d308-4852-b02f-ddb391f11127" containerName="payload-processing" containerID="cri-o://87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c" gracePeriod=25
Oct 30 22:03:10.211118 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.211098    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:03:10.228905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.228877    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc" podUID="7600b93e-60ad-4ac0-bc34-bdcc90047403" containerName="payload-pre-processing" containerID="cri-o://31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e" gracePeriod=25
Oct 30 22:03:10.321980 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.321954    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-metrics-certs\") pod \"ea05b385-c1b2-4020-839b-843de45e95bf\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") "
Oct 30 22:03:10.322125 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.322073    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea05b385-c1b2-4020-839b-843de45e95bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea05b385-c1b2-4020-839b-843de45e95bf-openshift-service-ca\") pod \"ea05b385-c1b2-4020-839b-843de45e95bf\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") "
Oct 30 22:03:10.322175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.322150    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-maas-api-tls\") pod \"ea05b385-c1b2-4020-839b-843de45e95bf\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") "
Oct 30 22:03:10.322244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.322229    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea05b385-c1b2-4020-839b-843de45e95bf-kube-api-access-lwszf\" (UniqueName: \"kubernetes.io/projected/ea05b385-c1b2-4020-839b-843de45e95bf-kube-api-access-lwszf\") pod \"ea05b385-c1b2-4020-839b-843de45e95bf\" (UID: \"ea05b385-c1b2-4020-839b-843de45e95bf\") "
Oct 30 22:03:10.322496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.322447    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea05b385-c1b2-4020-839b-843de45e95bf-openshift-service-ca" pod "ea05b385-c1b2-4020-839b-843de45e95bf" (UID: "ea05b385-c1b2-4020-839b-843de45e95bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:10.322962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.322942    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea05b385-c1b2-4020-839b-843de45e95bf-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.324191 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.324167    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-metrics-certs" pod "ea05b385-c1b2-4020-839b-843de45e95bf" (UID: "ea05b385-c1b2-4020-839b-843de45e95bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:10.324416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.324402    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-maas-api-tls" pod "ea05b385-c1b2-4020-839b-843de45e95bf" (UID: "ea05b385-c1b2-4020-839b-843de45e95bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:10.324499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.324459    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea05b385-c1b2-4020-839b-843de45e95bf-kube-api-access-lwszf" pod "ea05b385-c1b2-4020-839b-843de45e95bf" (UID: "ea05b385-c1b2-4020-839b-843de45e95bf"). InnerVolumeSpecName "kube-api-access-lwszf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:10.441542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.439338    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.441542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.439375    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwszf\" (UniqueName: \"kubernetes.io/projected/ea05b385-c1b2-4020-839b-843de45e95bf-kube-api-access-lwszf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.441542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.439391    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea05b385-c1b2-4020-839b-843de45e95bf-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.706724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.706655    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:03:10.761232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.761200    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:03:10.761416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.761392    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7600b93e-60ad-4ac0-bc34-bdcc90047403-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7600b93e-60ad-4ac0-bc34-bdcc90047403-plugins-config-volume\") pod \"7600b93e-60ad-4ac0-bc34-bdcc90047403\" (UID: \"7600b93e-60ad-4ac0-bc34-bdcc90047403\") "
Oct 30 22:03:10.761825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.761502    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7600b93e-60ad-4ac0-bc34-bdcc90047403-kube-api-access-mc9sf\" (UniqueName: \"kubernetes.io/projected/7600b93e-60ad-4ac0-bc34-bdcc90047403-kube-api-access-mc9sf\") pod \"7600b93e-60ad-4ac0-bc34-bdcc90047403\" (UID: \"7600b93e-60ad-4ac0-bc34-bdcc90047403\") "
Oct 30 22:03:10.761936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.761819    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7600b93e-60ad-4ac0-bc34-bdcc90047403-plugins-config-volume" pod "7600b93e-60ad-4ac0-bc34-bdcc90047403" (UID: "7600b93e-60ad-4ac0-bc34-bdcc90047403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:10.762617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.762594    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7600b93e-60ad-4ac0-bc34-bdcc90047403-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.764622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.764593    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7600b93e-60ad-4ac0-bc34-bdcc90047403-kube-api-access-mc9sf" pod "7600b93e-60ad-4ac0-bc34-bdcc90047403" (UID: "7600b93e-60ad-4ac0-bc34-bdcc90047403"). InnerVolumeSpecName "kube-api-access-mc9sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:10.863384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.863350    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e94194-d308-4852-b02f-ddb391f11127-kube-api-access-ggd7s\" (UniqueName: \"kubernetes.io/projected/a8e94194-d308-4852-b02f-ddb391f11127-kube-api-access-ggd7s\") pod \"a8e94194-d308-4852-b02f-ddb391f11127\" (UID: \"a8e94194-d308-4852-b02f-ddb391f11127\") "
Oct 30 22:03:10.863739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.863713    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e94194-d308-4852-b02f-ddb391f11127-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e94194-d308-4852-b02f-ddb391f11127-plugins-config-volume\") pod \"a8e94194-d308-4852-b02f-ddb391f11127\" (UID: \"a8e94194-d308-4852-b02f-ddb391f11127\") "
Oct 30 22:03:10.864134 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.864107    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e94194-d308-4852-b02f-ddb391f11127-plugins-config-volume" pod "a8e94194-d308-4852-b02f-ddb391f11127" (UID: "a8e94194-d308-4852-b02f-ddb391f11127"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:10.866386 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.866358    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e94194-d308-4852-b02f-ddb391f11127-kube-api-access-ggd7s" pod "a8e94194-d308-4852-b02f-ddb391f11127" (UID: "a8e94194-d308-4852-b02f-ddb391f11127"). InnerVolumeSpecName "kube-api-access-ggd7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:10.871146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.871122    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggd7s\" (UniqueName: \"kubernetes.io/projected/a8e94194-d308-4852-b02f-ddb391f11127-kube-api-access-ggd7s\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.871237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.871151    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9sf\" (UniqueName: \"kubernetes.io/projected/7600b93e-60ad-4ac0-bc34-bdcc90047403-kube-api-access-mc9sf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:10.871237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:10.871166    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e94194-d308-4852-b02f-ddb391f11127-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:11.179749 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.179706    4282 generic.go:356] "Generic (PLEG): container finished" podID="a8e94194-d308-4852-b02f-ddb391f11127" containerID="87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c" exitCode=0
Oct 30 22:03:11.180260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.179759    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"
Oct 30 22:03:11.180260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.179775    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5" event={"ID":"a8e94194-d308-4852-b02f-ddb391f11127","Type":"ContainerDied","Data":"87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c"}
Oct 30 22:03:11.180260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.179831    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5" event={"ID":"a8e94194-d308-4852-b02f-ddb391f11127","Type":"ContainerDied","Data":"f74e032071766ab77eaeacc7e242f4d6ea4614cc1d2041e8e0110cfb21f78141"}
Oct 30 22:03:11.180260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.179853    4282 scope.go:122] "RemoveContainer" containerID="87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c"
Oct 30 22:03:11.181545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.181525    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"
Oct 30 22:03:11.181802 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.181571    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2" event={"ID":"ea05b385-c1b2-4020-839b-843de45e95bf","Type":"ContainerDied","Data":"2ed0a1014cd2f65b5fb2b134ebff0c4d28ed781e8f944e68016dd30c3426d8ff"}
Oct 30 22:03:11.183528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.183438    4282 generic.go:356] "Generic (PLEG): container finished" podID="7600b93e-60ad-4ac0-bc34-bdcc90047403" containerID="31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e" exitCode=0
Oct 30 22:03:11.183528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.183511    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc" event={"ID":"7600b93e-60ad-4ac0-bc34-bdcc90047403","Type":"ContainerDied","Data":"31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e"}
Oct 30 22:03:11.183702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.183559    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc" event={"ID":"7600b93e-60ad-4ac0-bc34-bdcc90047403","Type":"ContainerDied","Data":"e7809be16dedfca241223261440514bc82e14737af820b485993776f8c87e096"}
Oct 30 22:03:11.183762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.183699    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"
Oct 30 22:03:11.211198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.211178    4282 scope.go:122] "RemoveContainer" containerID="87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c"
Oct 30 22:03:11.211506 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:11.211483    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c\": container with ID starting with 87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c not found: ID does not exist" containerID="87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c"
Oct 30 22:03:11.211573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.211517    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c"} err="failed to get container status \"87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c\": rpc error: code = NotFound desc = could not find container \"87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c\": container with ID starting with 87c8172d4b8bb586c80a0792885b0392f8c2e61e59c142c114f80a5767fa973c not found: ID does not exist"
Oct 30 22:03:11.211573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.211538    4282 scope.go:122] "RemoveContainer" containerID="b0744e3222314b4c3eb252c51c674da3fa670a811facaf762b003dee9d5c9a65"
Oct 30 22:03:11.223174 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.223165    4282 scope.go:122] "RemoveContainer" containerID="31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e"
Oct 30 22:03:11.227000 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.226978    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"]
Oct 30 22:03:11.230250 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.230229    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4cae46-6b7b5b69f-t6jpc"]
Oct 30 22:03:11.231489 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:11.231455    4282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e94194_d308_4852_b02f_ddb391f11127.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:03:11.233006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.232990    4282 scope.go:122] "RemoveContainer" containerID="31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e"
Oct 30 22:03:11.233322 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:11.233260    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e\": container with ID starting with 31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e not found: ID does not exist" containerID="31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e"
Oct 30 22:03:11.233322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.233294    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e"} err="failed to get container status \"31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e\": rpc error: code = NotFound desc = could not find container \"31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e\": container with ID starting with 31099dacfee549709f52eaaf79b89d1ffcc6d016f292ef1885d2cc443f15738e not found: ID does not exist"
Oct 30 22:03:11.235998 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.235977    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"]
Oct 30 22:03:11.240134 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.240115    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4cae46-578f5d9d6f-gnxn5"]
Oct 30 22:03:11.254452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.254435    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"]
Oct 30 22:03:11.258858 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.258839    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4cae46-784d867c6-ksdb2"]
Oct 30 22:03:11.515968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.515888    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7600b93e-60ad-4ac0-bc34-bdcc90047403" path="/var/lib/kubelet/pods/7600b93e-60ad-4ac0-bc34-bdcc90047403/volumes"
Oct 30 22:03:11.516570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.516551    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e94194-d308-4852-b02f-ddb391f11127" path="/var/lib/kubelet/pods/a8e94194-d308-4852-b02f-ddb391f11127/volumes"
Oct 30 22:03:11.516982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:11.516961    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea05b385-c1b2-4020-839b-843de45e95bf" path="/var/lib/kubelet/pods/ea05b385-c1b2-4020-839b-843de45e95bf/volumes"
Oct 30 22:03:12.680090 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:12.680056    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"]
Oct 30 22:03:12.681420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:12.680366    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l" podUID="f89a890d-368e-419c-8c6c-7ad5e0b78fe0" containerName="maas-api" containerID="cri-o://1a5ee8d361b97229406623bc7ef4384bde773a7b14b4ebc2eeb12b9d69892872" gracePeriod=30
Oct 30 22:03:12.895139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:12.894912    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb" podUID="83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" containerName="payload-processing" containerID="cri-o://421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87" gracePeriod=25
Oct 30 22:03:12.916508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:12.911538    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf" podUID="21d98e90-7393-4ad3-8411-1f2652c08ac2" containerName="payload-pre-processing" containerID="cri-o://95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a" gracePeriod=25
Oct 30 22:03:13.212500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.203502    4282 generic.go:356] "Generic (PLEG): container finished" podID="f89a890d-368e-419c-8c6c-7ad5e0b78fe0" containerID="1a5ee8d361b97229406623bc7ef4384bde773a7b14b4ebc2eeb12b9d69892872" exitCode=0
Oct 30 22:03:13.212500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.203609    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l" event={"ID":"f89a890d-368e-419c-8c6c-7ad5e0b78fe0","Type":"ContainerDied","Data":"1a5ee8d361b97229406623bc7ef4384bde773a7b14b4ebc2eeb12b9d69892872"}
Oct 30 22:03:13.371496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.367924    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:03:13.393657 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:03:13.391253    4282 logging.go:55] [core] [Channel #509 SubChannel #510]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.136:50052", ServerName: "10.133.0.136:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.136:50052: connect: connection refused"
Oct 30 22:03:13.473863 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.473834    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls\") pod \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") "
Oct 30 22:03:13.474323 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.474299    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-openshift-service-ca\") pod \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") "
Oct 30 22:03:13.474610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.474592    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-kube-api-access-8h5gx\" (UniqueName: \"kubernetes.io/projected/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-kube-api-access-8h5gx\") pod \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") "
Oct 30 22:03:13.474823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.474807    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-metrics-certs\") pod \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\" (UID: \"f89a890d-368e-419c-8c6c-7ad5e0b78fe0\") "
Oct 30 22:03:13.475675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.475653    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-openshift-service-ca" pod "f89a890d-368e-419c-8c6c-7ad5e0b78fe0" (UID: "f89a890d-368e-419c-8c6c-7ad5e0b78fe0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:13.476375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.476357    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:13.479523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.478784    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls" pod "f89a890d-368e-419c-8c6c-7ad5e0b78fe0" (UID: "f89a890d-368e-419c-8c6c-7ad5e0b78fe0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:13.491131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.491056    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-kube-api-access-8h5gx" pod "f89a890d-368e-419c-8c6c-7ad5e0b78fe0" (UID: "f89a890d-368e-419c-8c6c-7ad5e0b78fe0"). InnerVolumeSpecName "kube-api-access-8h5gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:13.492744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.492717    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-metrics-certs" pod "f89a890d-368e-419c-8c6c-7ad5e0b78fe0" (UID: "f89a890d-368e-419c-8c6c-7ad5e0b78fe0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:13.577859 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.577826    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5gx\" (UniqueName: \"kubernetes.io/projected/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-kube-api-access-8h5gx\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:13.577999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.577865    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:13.577999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.577879    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f89a890d-368e-419c-8c6c-7ad5e0b78fe0-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:13.607023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.606994    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:03:13.788536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.787175    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d98e90-7393-4ad3-8411-1f2652c08ac2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d98e90-7393-4ad3-8411-1f2652c08ac2-plugins-config-volume\") pod \"21d98e90-7393-4ad3-8411-1f2652c08ac2\" (UID: \"21d98e90-7393-4ad3-8411-1f2652c08ac2\") "
Oct 30 22:03:13.788536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.787396    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d98e90-7393-4ad3-8411-1f2652c08ac2-kube-api-access-6l87q\" (UniqueName: \"kubernetes.io/projected/21d98e90-7393-4ad3-8411-1f2652c08ac2-kube-api-access-6l87q\") pod \"21d98e90-7393-4ad3-8411-1f2652c08ac2\" (UID: \"21d98e90-7393-4ad3-8411-1f2652c08ac2\") "
Oct 30 22:03:13.789068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.788763    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d98e90-7393-4ad3-8411-1f2652c08ac2-plugins-config-volume" pod "21d98e90-7393-4ad3-8411-1f2652c08ac2" (UID: "21d98e90-7393-4ad3-8411-1f2652c08ac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:13.877793 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.877752    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d98e90-7393-4ad3-8411-1f2652c08ac2-kube-api-access-6l87q" pod "21d98e90-7393-4ad3-8411-1f2652c08ac2" (UID: "21d98e90-7393-4ad3-8411-1f2652c08ac2"). InnerVolumeSpecName "kube-api-access-6l87q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:13.884640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.884609    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:03:13.889679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.889652    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d98e90-7393-4ad3-8411-1f2652c08ac2-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:13.894907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.894879    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l87q\" (UniqueName: \"kubernetes.io/projected/21d98e90-7393-4ad3-8411-1f2652c08ac2-kube-api-access-6l87q\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:13.996514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.996485    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-kube-api-access-7n49m\" (UniqueName: \"kubernetes.io/projected/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-kube-api-access-7n49m\") pod \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\" (UID: \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\") "
Oct 30 22:03:13.996715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.996549    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-plugins-config-volume\") pod \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\" (UID: \"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d\") "
Oct 30 22:03:13.996942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.996917    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-plugins-config-volume" pod "83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" (UID: "83fa41ca-03bc-4817-a38c-ac1c9cc1d10d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:13.997248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.997229    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:13.998823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:13.998800    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-kube-api-access-7n49m" pod "83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" (UID: "83fa41ca-03bc-4817-a38c-ac1c9cc1d10d"). InnerVolumeSpecName "kube-api-access-7n49m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:14.107544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.098832    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n49m\" (UniqueName: \"kubernetes.io/projected/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d-kube-api-access-7n49m\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:14.258503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.254133    4282 generic.go:356] "Generic (PLEG): container finished" podID="83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" containerID="421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87" exitCode=0
Oct 30 22:03:14.258503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.254234    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb" event={"ID":"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d","Type":"ContainerDied","Data":"421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87"}
Oct 30 22:03:14.258503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.254281    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb" event={"ID":"83fa41ca-03bc-4817-a38c-ac1c9cc1d10d","Type":"ContainerDied","Data":"9f9b4ddfb3e0118fdb66af203dbf9b53db5d11ed0acac6da8604da3246956f4a"}
Oct 30 22:03:14.258503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.254307    4282 scope.go:122] "RemoveContainer" containerID="421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87"
Oct 30 22:03:14.258503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.254445    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"
Oct 30 22:03:14.280496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.279416    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"
Oct 30 22:03:14.307508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.279275    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l" event={"ID":"f89a890d-368e-419c-8c6c-7ad5e0b78fe0","Type":"ContainerDied","Data":"c33b188ee9f6634c7d1994b606334f5887f48aa282bb39217a11ea9375de3b09"}
Oct 30 22:03:14.323591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.323564    4282 generic.go:356] "Generic (PLEG): container finished" podID="21d98e90-7393-4ad3-8411-1f2652c08ac2" containerID="95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a" exitCode=0
Oct 30 22:03:14.323718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.323604    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf" event={"ID":"21d98e90-7393-4ad3-8411-1f2652c08ac2","Type":"ContainerDied","Data":"95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a"}
Oct 30 22:03:14.323718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.323637    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf" event={"ID":"21d98e90-7393-4ad3-8411-1f2652c08ac2","Type":"ContainerDied","Data":"e8001f8463154e5d1016c020058bfe97fdee08c965f1d8ea504f4a8eba696d12"}
Oct 30 22:03:14.323718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.323692    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"
Oct 30 22:03:14.330078 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.329730    4282 scope.go:122] "RemoveContainer" containerID="421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87"
Oct 30 22:03:14.330765 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:14.330350    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87\": container with ID starting with 421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87 not found: ID does not exist" containerID="421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87"
Oct 30 22:03:14.330765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.330389    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87"} err="failed to get container status \"421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87\": rpc error: code = NotFound desc = could not find container \"421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87\": container with ID starting with 421e3d72e80986da8b0b11ed888b34fe5bafaff06136a9f50f86ae1976c11f87 not found: ID does not exist"
Oct 30 22:03:14.330765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.330413    4282 scope.go:122] "RemoveContainer" containerID="1a5ee8d361b97229406623bc7ef4384bde773a7b14b4ebc2eeb12b9d69892872"
Oct 30 22:03:14.344962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.344929    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"]
Oct 30 22:03:14.419493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.401905    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb" podUID="83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.136:50052\" within 1s: context deadline exceeded"
Oct 30 22:03:14.464496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.456782    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bca8c9-65b6c9b685-sp6gb"]
Oct 30 22:03:14.562491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.561502    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"]
Oct 30 22:03:14.599497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.597588    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bca8c9-7df557cc6d-t6t5l"]
Oct 30 22:03:14.645711 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.639905    4282 scope.go:122] "RemoveContainer" containerID="95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a"
Oct 30 22:03:14.650504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.645818    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"]
Oct 30 22:03:14.703557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.699548    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bca8c9-56557fb66c-lmplf"]
Oct 30 22:03:14.804119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.804066    4282 scope.go:122] "RemoveContainer" containerID="95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a"
Oct 30 22:03:14.804599 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:14.804522    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a\": container with ID starting with 95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a not found: ID does not exist" containerID="95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a"
Oct 30 22:03:14.804599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:14.804558    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a"} err="failed to get container status \"95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a\": rpc error: code = NotFound desc = could not find container \"95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a\": container with ID starting with 95a51ab6317afef59519bf5322625ab05f9074f7e1c68c61070795a5d67c7d7a not found: ID does not exist"
Oct 30 22:03:15.241697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:15.241665    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:03:16.123612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.104608    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d98e90-7393-4ad3-8411-1f2652c08ac2" path="/var/lib/kubelet/pods/21d98e90-7393-4ad3-8411-1f2652c08ac2/volumes"
Oct 30 22:03:16.359503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.355243    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:16.504502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.479281    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" path="/var/lib/kubelet/pods/83fa41ca-03bc-4817-a38c-ac1c9cc1d10d/volumes"
Oct 30 22:03:16.530604 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.505534    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89a890d-368e-419c-8c6c-7ad5e0b78fe0" path="/var/lib/kubelet/pods/f89a890d-368e-419c-8c6c-7ad5e0b78fe0/volumes"
Oct 30 22:03:16.552600 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:16.531371    4282 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.047s"
Oct 30 22:03:16.752660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.752628    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:03:16.754030 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.754008    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:03:16.757766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.503294    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:16.773487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.620635    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:16.773487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.499425    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:16.773487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.503325    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:17.002991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:16.983252    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:21.243372 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:21.243333    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"]
Oct 30 22:03:21.243952 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:21.243636    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" containerID="cri-o://8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0" gracePeriod=30
Oct 30 22:03:22.150805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:22.150768    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:03:22.150964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:22.150840    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:03:23.217498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.217437    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"]
Oct 30 22:03:23.218050 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218030    4282 cpu_manager.go:414] "containerMap: removing container" podUID="21d98e90-7393-4ad3-8411-1f2652c08ac2" containerName="payload-pre-processing"
Oct 30 22:03:23.218050 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218048    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d98e90-7393-4ad3-8411-1f2652c08ac2" containerName="payload-pre-processing"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218079    4282 cpu_manager.go:414] "containerMap: removing container" podUID="83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" containerName="payload-processing"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218084    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" containerName="payload-processing"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218093    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7600b93e-60ad-4ac0-bc34-bdcc90047403" containerName="payload-pre-processing"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218098    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7600b93e-60ad-4ac0-bc34-bdcc90047403" containerName="payload-pre-processing"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218104    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f89a890d-368e-419c-8c6c-7ad5e0b78fe0" containerName="maas-api"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218110    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89a890d-368e-419c-8c6c-7ad5e0b78fe0" containerName="maas-api"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218130    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ea05b385-c1b2-4020-839b-843de45e95bf" containerName="maas-api"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218135    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea05b385-c1b2-4020-839b-843de45e95bf" containerName="maas-api"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218145    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a8e94194-d308-4852-b02f-ddb391f11127" containerName="payload-processing"
Oct 30 22:03:23.218221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218149    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e94194-d308-4852-b02f-ddb391f11127" containerName="payload-processing"
Oct 30 22:03:23.218640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218228    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fa41ca-03bc-4817-a38c-ac1c9cc1d10d" containerName="payload-processing"
Oct 30 22:03:23.218640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218240    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7600b93e-60ad-4ac0-bc34-bdcc90047403" containerName="payload-pre-processing"
Oct 30 22:03:23.218640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218248    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89a890d-368e-419c-8c6c-7ad5e0b78fe0" containerName="maas-api"
Oct 30 22:03:23.218640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218262    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea05b385-c1b2-4020-839b-843de45e95bf" containerName="maas-api"
Oct 30 22:03:23.218640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218273    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e94194-d308-4852-b02f-ddb391f11127" containerName="payload-processing"
Oct 30 22:03:23.218640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.218285    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d98e90-7393-4ad3-8411-1f2652c08ac2" containerName="payload-pre-processing"
Oct 30 22:03:23.223705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.223684    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.244810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.244782    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2cfa71-openshift-default-dockercfg-wbbdt\""
Oct 30 22:03:23.246341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.246312    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"]
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.277694    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-envoy\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.277770    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-socket\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.277872    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-credential-socket\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.277910    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqfg\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-kube-api-access-gsqfg\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.277977    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-istio-token\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.278010    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822606a9-93cc-468f-99d6-6a040df39808-istio-podinfo\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.278089    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-data\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.278145    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-certs\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.293329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.278302    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822606a9-93cc-468f-99d6-6a040df39808-istiod-ca-cert\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.398676    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-certs\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.398848    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822606a9-93cc-468f-99d6-6a040df39808-istiod-ca-cert\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.398901    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-envoy\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.398932    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-socket\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.399009    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-credential-socket\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.399045    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqfg\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-kube-api-access-gsqfg\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.399119    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-istio-token\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.399151    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822606a9-93cc-468f-99d6-6a040df39808-istio-podinfo\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.399245    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-data\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.399714    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-data\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.399956    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-certs\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.418505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.400550    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822606a9-93cc-468f-99d6-6a040df39808-istiod-ca-cert\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.436528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.423939    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-credential-socket\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.446585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.446545    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-socket\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.492611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.477933    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-envoy\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.517526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.501996    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-istio-token\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.517526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.505579    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqfg\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-kube-api-access-gsqfg\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.572501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.566081    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822606a9-93cc-468f-99d6-6a040df39808-istio-podinfo\") pod \"e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") " pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.840856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.840764    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:23.887013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.886983    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"]
Oct 30 22:03:23.892329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.892306    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.915826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.915763    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.915972 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.915859    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz88d\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-kube-api-access-jz88d\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.916039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.916006    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.916093 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.916056    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.916145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.916128    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f55e61-ab06-4d34-97e7-2d94331f99ce-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.926818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.916204    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.926818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.916249    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.926818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.916413    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.926818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.916654    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:23.933072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:23.932824    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"]
Oct 30 22:03:24.018103 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018068    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f55e61-ab06-4d34-97e7-2d94331f99ce-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018134    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018165    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018220    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018257    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018330    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018385    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz88d\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-kube-api-access-jz88d\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018448    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018503    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018877    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.018965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.018917    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f55e61-ab06-4d34-97e7-2d94331f99ce-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.019154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.019132    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.019238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.019218    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.019355 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.019327    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.021279 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.021256    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.021925 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.021901    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.032652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.032619    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.033643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.033621    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz88d\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-kube-api-access-jz88d\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.160397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.160372    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"]
Oct 30 22:03:24.163687 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:03:24.163655    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822606a9_93cc_468f_99d6_6a040df39808.slice/crio-4ca1188453c3a36c147c02af60ba8d47cdaf12fa338b9d0d063daf113461fb3b WatchSource:0}: Error finding container 4ca1188453c3a36c147c02af60ba8d47cdaf12fa338b9d0d063daf113461fb3b: Status 404 returned error can't find the container with id 4ca1188453c3a36c147c02af60ba8d47cdaf12fa338b9d0d063daf113461fb3b
Oct 30 22:03:24.166819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.166784    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:24.166967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.166862    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:24.166967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.166905    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:24.212433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.212408    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:24.516133 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.516105    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"]
Oct 30 22:03:24.518025 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:03:24.517995    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f55e61_ab06_4d34_97e7_2d94331f99ce.slice/crio-70e47d032555db36cd6c017a15bd8151b47114a25831e75ffb051421343665c7 WatchSource:0}: Error finding container 70e47d032555db36cd6c017a15bd8151b47114a25831e75ffb051421343665c7: Status 404 returned error can't find the container with id 70e47d032555db36cd6c017a15bd8151b47114a25831e75ffb051421343665c7
Oct 30 22:03:24.521375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.521334    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:24.521504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.521430    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:24.521560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.521514    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:24.634253 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.634215    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" event={"ID":"73f55e61-ab06-4d34-97e7-2d94331f99ce","Type":"ContainerStarted","Data":"70e47d032555db36cd6c017a15bd8151b47114a25831e75ffb051421343665c7"}
Oct 30 22:03:24.640874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.640831    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" event={"ID":"822606a9-93cc-468f-99d6-6a040df39808","Type":"ContainerStarted","Data":"37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362"}
Oct 30 22:03:24.641164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.641134    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" event={"ID":"822606a9-93cc-468f-99d6-6a040df39808","Type":"ContainerStarted","Data":"4ca1188453c3a36c147c02af60ba8d47cdaf12fa338b9d0d063daf113461fb3b"}
Oct 30 22:03:24.752539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.752450    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podStartSLOduration=1.7524297930000001 podStartE2EDuration="1.752429793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:24.749286887 +0000 UTC m=+1423.407383111" watchObservedRunningTime="2026-10-30 22:03:24.752429793 +0000 UTC m=+1423.410526019"
Oct 30 22:03:24.841632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:24.841528    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:25.686496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:25.685780    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" event={"ID":"73f55e61-ab06-4d34-97e7-2d94331f99ce","Type":"ContainerStarted","Data":"9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4"}
Oct 30 22:03:25.711406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:25.711355    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podStartSLOduration=2.711341741 podStartE2EDuration="2.711341741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:25.708017236 +0000 UTC m=+1424.366113460" watchObservedRunningTime="2026-10-30 22:03:25.711341741 +0000 UTC m=+1424.369437964"
Oct 30 22:03:25.847400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:25.847371    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:26.212686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.212601    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:26.214010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.213987    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 22:03:26.214129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.214058    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 22:03:26.696957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.696924    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:03:26.701321 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.701294    4282 generic.go:356] "Generic (PLEG): container finished" podID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerID="8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0" exitCode=0
Oct 30 22:03:26.701674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.701651    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" event={"ID":"9eda7d94-5787-4a6d-97ce-bd2337998a88","Type":"ContainerDied","Data":"8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0"}
Oct 30 22:03:26.701880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.701862    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g" event={"ID":"9eda7d94-5787-4a6d-97ce-bd2337998a88","Type":"ContainerDied","Data":"e537eda573b6d23c51d7fa221f96ed91975970be243d80c26145a5f59f29d5f8"}
Oct 30 22:03:26.702012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.701998    4282 scope.go:122] "RemoveContainer" containerID="8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0"
Oct 30 22:03:26.709940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.709722    4282 generic.go:356] "Generic (PLEG): container finished" podID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerID="0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d" exitCode=0
Oct 30 22:03:26.710413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.710024    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" event={"ID":"b0bb00c7-522d-4e8a-baf0-95897d40452c","Type":"ContainerDied","Data":"0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d"}
Oct 30 22:03:26.710647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.710584    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:26.711231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.711161    4282 scope.go:122] "RemoveContainer" containerID="0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d"
Oct 30 22:03:26.711837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.711795    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:03:26.718720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.718694    4282 scope.go:122] "RemoveContainer" containerID="56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81"
Oct 30 22:03:26.719074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.719034    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:26.719167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.719130    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:26.719237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.719175    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:26.742028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.742011    4282 scope.go:122] "RemoveContainer" containerID="8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0"
Oct 30 22:03:26.742848 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:26.742704    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0\": container with ID starting with 8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0 not found: ID does not exist" containerID="8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0"
Oct 30 22:03:26.742848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.742745    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0"} err="failed to get container status \"8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0\": rpc error: code = NotFound desc = could not find container \"8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0\": container with ID starting with 8478e2bc5f15138c4c7ca301e65081c46f79d2ede417268083bdc912255114c0 not found: ID does not exist"
Oct 30 22:03:26.742848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.742769    4282 scope.go:122] "RemoveContainer" containerID="56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81"
Oct 30 22:03:26.745274 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:26.745241    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81\": container with ID starting with 56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81 not found: ID does not exist" containerID="56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81"
Oct 30 22:03:26.745369 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.745283    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81"} err="failed to get container status \"56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81\": rpc error: code = NotFound desc = could not find container \"56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81\": container with ID starting with 56390dcbe35e1dc4532b3ce6fd60a2d9d7d6978baba01f4fae7e34044d9a3c81 not found: ID does not exist"
Oct 30 22:03:26.801778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.801747    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-kube-api-access-mk4n8\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-kube-api-access-mk4n8\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.801934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.801796    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eda7d94-5787-4a6d-97ce-bd2337998a88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eda7d94-5787-4a6d-97ce-bd2337998a88-istiod-ca-cert\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.801934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.801847    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-podinfo\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.801934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.801912    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-token\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-token\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.802115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.801953    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-credential-socket\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.802115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.801969    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-certs\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.802115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.802007    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-socket\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.802288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.802267    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eda7d94-5787-4a6d-97ce-bd2337998a88-istiod-ca-cert" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:26.802619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.802584    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-credential-socket" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:26.802853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.802817    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-certs" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:26.803059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.803036    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-socket" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:26.803218 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.803199    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-envoy\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.803668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.803652    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-data\") pod \"9eda7d94-5787-4a6d-97ce-bd2337998a88\" (UID: \"9eda7d94-5787-4a6d-97ce-bd2337998a88\") "
Oct 30 22:03:26.804110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.804000    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-data" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:26.805261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.805225    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-kube-api-access-mk4n8" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "kube-api-access-mk4n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:26.805608 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.805446    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-token" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:26.805703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.805666    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-podinfo" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:26.807058 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807025    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-envoy" pod "9eda7d94-5787-4a6d-97ce-bd2337998a88" (UID: "9eda7d94-5787-4a6d-97ce-bd2337998a88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:26.807162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807069    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.807162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807093    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4n8\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-kube-api-access-mk4n8\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.807162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807108    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eda7d94-5787-4a6d-97ce-bd2337998a88-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.807162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807149    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.807394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807162    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.807394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807175    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.807394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807189    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.807394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.807227    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:26.908834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:26.908809    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eda7d94-5787-4a6d-97ce-bd2337998a88-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:27.212822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:27.212788    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 22:03:27.212999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:27.212836    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 22:03:27.717804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:27.717769    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"
Oct 30 22:03:27.721311 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:27.720863    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" event={"ID":"b0bb00c7-522d-4e8a-baf0-95897d40452c","Type":"ContainerStarted","Data":"cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b"}
Oct 30 22:03:27.805636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:27.805601    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"]
Oct 30 22:03:27.810558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:27.810533    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bca8c9-openshift-default-69dd8974cc-9b72g"]
Oct 30 22:03:28.215569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.215531    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 22:03:28.215789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.215588    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 22:03:28.355860 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.349841    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:03:28.380499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.361313    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:03:28.380499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.361371    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 22:03:28.615538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.613393    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"]
Oct 30 22:03:28.615538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.614372    4282 cpu_manager.go:414] "containerMap: removing container" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy"
Oct 30 22:03:28.615538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.614394    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy"
Oct 30 22:03:28.615538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.614418    4282 cpu_manager.go:414] "containerMap: removing container" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy"
Oct 30 22:03:28.615538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.614427    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy"
Oct 30 22:03:28.615538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.614646    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy"
Oct 30 22:03:28.615538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.614665    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" containerName="istio-proxy"
Oct 30 22:03:28.619434 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.619411    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"]
Oct 30 22:03:28.619773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.619709    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:28.694795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.694767    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2cfa71\""
Oct 30 22:03:28.695162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.695145    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2cfa71-dockercfg-2f8j5\""
Oct 30 22:03:28.857547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.836249    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe436f-d58a-45d8-a33c-23d32e87f673-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl\" (UID: \"dcbe436f-d58a-45d8-a33c-23d32e87f673\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:28.885905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.885794    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmm79\" (UniqueName: \"kubernetes.io/projected/dcbe436f-d58a-45d8-a33c-23d32e87f673-kube-api-access-qmm79\") pod \"payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl\" (UID: \"dcbe436f-d58a-45d8-a33c-23d32e87f673\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:28.945496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.943905    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"]
Oct 30 22:03:28.945496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.944148    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs" podUID="b0905a38-269b-4f46-be5b-aca1ed6f954c" containerName="maas-api" containerID="cri-o://d6e60be2c88bc0bdbb24caca11369ad30d4223cf2e307e96006e7f8fd85194da" gracePeriod=30
Oct 30 22:03:29.008524 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.997910    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmm79\" (UniqueName: \"kubernetes.io/projected/dcbe436f-d58a-45d8-a33c-23d32e87f673-kube-api-access-qmm79\") pod \"payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl\" (UID: \"dcbe436f-d58a-45d8-a33c-23d32e87f673\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:29.008524 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:28.998137    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe436f-d58a-45d8-a33c-23d32e87f673-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl\" (UID: \"dcbe436f-d58a-45d8-a33c-23d32e87f673\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:29.060515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.023227    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe436f-d58a-45d8-a33c-23d32e87f673-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl\" (UID: \"dcbe436f-d58a-45d8-a33c-23d32e87f673\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:29.269412 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.257660    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 22:03:29.284545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.280236    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmm79\" (UniqueName: \"kubernetes.io/projected/dcbe436f-d58a-45d8-a33c-23d32e87f673-kube-api-access-qmm79\") pod \"payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl\" (UID: \"dcbe436f-d58a-45d8-a33c-23d32e87f673\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:29.333621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.261512    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 22:03:29.485493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.458063    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:03:29.485493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.458185    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 22:03:29.685411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.685369    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs" podUID="b0905a38-269b-4f46-be5b-aca1ed6f954c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.144:8443/health\": dial tcp 10.133.0.144:8443: connect: connection refused"
Oct 30 22:03:29.780499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.780422    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:29.821392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.821341    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eda7d94-5787-4a6d-97ce-bd2337998a88" path="/var/lib/kubelet/pods/9eda7d94-5787-4a6d-97ce-bd2337998a88/volumes"
Oct 30 22:03:29.822110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.822076    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"]
Oct 30 22:03:29.823788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.823767    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"]
Oct 30 22:03:29.919538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.919498    4282 generic.go:356] "Generic (PLEG): container finished" podID="b0905a38-269b-4f46-be5b-aca1ed6f954c" containerID="d6e60be2c88bc0bdbb24caca11369ad30d4223cf2e307e96006e7f8fd85194da" exitCode=0
Oct 30 22:03:29.920166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:29.920137    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs" event={"ID":"b0905a38-269b-4f46-be5b-aca1ed6f954c","Type":"ContainerDied","Data":"d6e60be2c88bc0bdbb24caca11369ad30d4223cf2e307e96006e7f8fd85194da"}
Oct 30 22:03:30.212254 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.212225    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"]
Oct 30 22:03:30.214189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.213812    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 22:03:30.214189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.213857    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 22:03:30.370805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.370337    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:03:30.370805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.370395    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 22:03:30.395508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.393177    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:03:30.438560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.432433    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0905a38-269b-4f46-be5b-aca1ed6f954c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0905a38-269b-4f46-be5b-aca1ed6f954c-openshift-service-ca\") pod \"b0905a38-269b-4f46-be5b-aca1ed6f954c\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") "
Oct 30 22:03:30.438560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.432641    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-maas-api-tls\") pod \"b0905a38-269b-4f46-be5b-aca1ed6f954c\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") "
Oct 30 22:03:30.438560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.432674    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-metrics-certs\") pod \"b0905a38-269b-4f46-be5b-aca1ed6f954c\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") "
Oct 30 22:03:30.438560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.432850    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0905a38-269b-4f46-be5b-aca1ed6f954c-kube-api-access-q7vjv\" (UniqueName: \"kubernetes.io/projected/b0905a38-269b-4f46-be5b-aca1ed6f954c-kube-api-access-q7vjv\") pod \"b0905a38-269b-4f46-be5b-aca1ed6f954c\" (UID: \"b0905a38-269b-4f46-be5b-aca1ed6f954c\") "
Oct 30 22:03:30.438560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.434499    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0905a38-269b-4f46-be5b-aca1ed6f954c-openshift-service-ca" pod "b0905a38-269b-4f46-be5b-aca1ed6f954c" (UID: "b0905a38-269b-4f46-be5b-aca1ed6f954c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:30.441538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.441507    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-metrics-certs" pod "b0905a38-269b-4f46-be5b-aca1ed6f954c" (UID: "b0905a38-269b-4f46-be5b-aca1ed6f954c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:30.442762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.442730    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0905a38-269b-4f46-be5b-aca1ed6f954c-kube-api-access-q7vjv" pod "b0905a38-269b-4f46-be5b-aca1ed6f954c" (UID: "b0905a38-269b-4f46-be5b-aca1ed6f954c"). InnerVolumeSpecName "kube-api-access-q7vjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:30.444291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.443150    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-maas-api-tls" pod "b0905a38-269b-4f46-be5b-aca1ed6f954c" (UID: "b0905a38-269b-4f46-be5b-aca1ed6f954c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:30.535260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.535227    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0905a38-269b-4f46-be5b-aca1ed6f954c-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:30.535260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.535261    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:30.535502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.535276    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0905a38-269b-4f46-be5b-aca1ed6f954c-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:30.535502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.535291    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7vjv\" (UniqueName: \"kubernetes.io/projected/b0905a38-269b-4f46-be5b-aca1ed6f954c-kube-api-access-q7vjv\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:30.939029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.938990    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl" event={"ID":"dcbe436f-d58a-45d8-a33c-23d32e87f673","Type":"ContainerStarted","Data":"c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6"}
Oct 30 22:03:30.939455 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.939049    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl" event={"ID":"dcbe436f-d58a-45d8-a33c-23d32e87f673","Type":"ContainerStarted","Data":"f00a64636ef3d55563062f86534b7cfd919b7370b2509978013929601a34bb5d"}
Oct 30 22:03:30.939455 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.939100    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:30.940651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.940630    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"
Oct 30 22:03:30.940760 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.940630    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs" event={"ID":"b0905a38-269b-4f46-be5b-aca1ed6f954c","Type":"ContainerDied","Data":"d206659394605115af00cc96ccddc576eb18ca5a6c9557a85ca69c4d311f4213"}
Oct 30 22:03:30.940817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:30.940794    4282 scope.go:122] "RemoveContainer" containerID="d6e60be2c88bc0bdbb24caca11369ad30d4223cf2e307e96006e7f8fd85194da"
Oct 30 22:03:31.019497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:31.019430    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl" podStartSLOduration=3.019418017 podStartE2EDuration="3.019418017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:31.018543445 +0000 UTC m=+1429.676639671" watchObservedRunningTime="2026-10-30 22:03:31.019418017 +0000 UTC m=+1429.677514239"
Oct 30 22:03:31.037948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:31.037919    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"]
Oct 30 22:03:31.044731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:31.044707    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-90b6f5-7788bff64b-kmjcs"]
Oct 30 22:03:31.224549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:31.215856    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 22:03:31.224549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:31.215912    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 22:03:31.350312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:31.350282    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:03:31.350622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:31.350596    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 22:03:31.488364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:31.488271    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0905a38-269b-4f46-be5b-aca1ed6f954c" path="/var/lib/kubelet/pods/b0905a38-269b-4f46-be5b-aca1ed6f954c/volumes"
Oct 30 22:03:32.213102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.213064    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 22:03:32.213523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.213133    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 22:03:32.346865 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.346829    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:03:32.347038 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.346894    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 22:03:32.628851 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.628220    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"]
Oct 30 22:03:32.629212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.629186    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b0905a38-269b-4f46-be5b-aca1ed6f954c" containerName="maas-api"
Oct 30 22:03:32.629212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.629213    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0905a38-269b-4f46-be5b-aca1ed6f954c" containerName="maas-api"
Oct 30 22:03:32.629421 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.629407    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0905a38-269b-4f46-be5b-aca1ed6f954c" containerName="maas-api"
Oct 30 22:03:32.634896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.634877    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.642577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.642385    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-2cfa71-serving-cert\""
Oct 30 22:03:32.644413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.644387    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"]
Oct 30 22:03:32.708215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.708184    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2d328-8340-4a8c-ace4-f1525004c794-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.733824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.733783    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xg8n\" (UniqueName: \"kubernetes.io/projected/f9b2d328-8340-4a8c-ace4-f1525004c794-kube-api-access-8xg8n\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.734019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.733893    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.734019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.733925    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-metrics-certs\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.836627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.836567    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2d328-8340-4a8c-ace4-f1525004c794-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.876722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.836791    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xg8n\" (UniqueName: \"kubernetes.io/projected/f9b2d328-8340-4a8c-ace4-f1525004c794-kube-api-access-8xg8n\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.876722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.836912    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.876722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.836943    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-metrics-certs\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.876722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.848349    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2d328-8340-4a8c-ace4-f1525004c794-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.883157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.883073    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-metrics-certs\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.906345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.906312    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xg8n\" (UniqueName: \"kubernetes.io/projected/f9b2d328-8340-4a8c-ace4-f1525004c794-kube-api-access-8xg8n\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.909351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.909318    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:32.976801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:32.976757    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:33.091151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:33.091115    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:03:33.354520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:33.352165    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 22:03:33.354520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:33.352223    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 22:03:33.585174 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:33.585100    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"]
Oct 30 22:03:33.589052 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:03:33.589022    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b2d328_8340_4a8c_ace4_f1525004c794.slice/crio-27226e588a71bdeb0a483ea01fba303c0e7b889edc6dbf4c3e572d7f265386c7 WatchSource:0}: Error finding container 27226e588a71bdeb0a483ea01fba303c0e7b889edc6dbf4c3e572d7f265386c7: Status 404 returned error can't find the container with id 27226e588a71bdeb0a483ea01fba303c0e7b889edc6dbf4c3e572d7f265386c7
Oct 30 22:03:33.601922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:33.601900    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:33.602306 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:33.602286    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:33.603161 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:33.603139    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:03:34.052337 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:34.052300    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9" event={"ID":"f9b2d328-8340-4a8c-ace4-f1525004c794","Type":"ContainerStarted","Data":"27226e588a71bdeb0a483ea01fba303c0e7b889edc6dbf4c3e572d7f265386c7"}
Oct 30 22:03:34.314239 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:34.314162    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:03:34.351265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:34.351238    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:03:34.352579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:34.352553    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:03:34.661188 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:34.661097    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:03:34.822887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:34.822847    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc" podUID="8d984bb8-3ed8-4790-827a-4292af0cfcd2" containerName="payload-processing" containerID="cri-o://11f7ae5ed6c8d52371cd85c53a71551950d4d8bd798268ec298304281a07db11" gracePeriod=25
Oct 30 22:03:34.824364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:34.824319    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48" podUID="501a2c31-fb1e-47b9-acf7-aff6367ef3ba" containerName="payload-pre-processing" containerID="cri-o://ae96f8ae3b174af0137035b97c659a45c4bbbbdf71d538fbef3df5064ac6d0ad" gracePeriod=25
Oct 30 22:03:35.115378 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.115046    4282 generic.go:356] "Generic (PLEG): container finished" podID="501a2c31-fb1e-47b9-acf7-aff6367ef3ba" containerID="ae96f8ae3b174af0137035b97c659a45c4bbbbdf71d538fbef3df5064ac6d0ad" exitCode=0
Oct 30 22:03:35.115378 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.115301    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48" event={"ID":"501a2c31-fb1e-47b9-acf7-aff6367ef3ba","Type":"ContainerDied","Data":"ae96f8ae3b174af0137035b97c659a45c4bbbbdf71d538fbef3df5064ac6d0ad"}
Oct 30 22:03:35.119299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.119124    4282 generic.go:356] "Generic (PLEG): container finished" podID="8d984bb8-3ed8-4790-827a-4292af0cfcd2" containerID="11f7ae5ed6c8d52371cd85c53a71551950d4d8bd798268ec298304281a07db11" exitCode=0
Oct 30 22:03:35.119299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.119212    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc" event={"ID":"8d984bb8-3ed8-4790-827a-4292af0cfcd2","Type":"ContainerDied","Data":"11f7ae5ed6c8d52371cd85c53a71551950d4d8bd798268ec298304281a07db11"}
Oct 30 22:03:35.126151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.125278    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9" event={"ID":"f9b2d328-8340-4a8c-ace4-f1525004c794","Type":"ContainerStarted","Data":"47327a57bb8d0158054070eff1c9afc3cfca6f8d39fff7cb0a77df372c884336"}
Oct 30 22:03:35.126151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.125952    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:35.147318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.145503    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9" podStartSLOduration=2.676778788 podStartE2EDuration="3.145482822s" totalImagesPullingTime="468.704034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:35.143604213 +0000 UTC m=+1433.801700437" watchObservedRunningTime="2026-10-30 22:03:35.145482822 +0000 UTC m=+1433.803579049"
Oct 30 22:03:35.349052 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.343033    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:03:35.454982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.454953    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-plugins-config-volume\") pod \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\" (UID: \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\") "
Oct 30 22:03:35.455322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.455302    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-kube-api-access-ztrjh\" (UniqueName: \"kubernetes.io/projected/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-kube-api-access-ztrjh\") pod \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\" (UID: \"501a2c31-fb1e-47b9-acf7-aff6367ef3ba\") "
Oct 30 22:03:35.457265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.457243    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-plugins-config-volume" pod "501a2c31-fb1e-47b9-acf7-aff6367ef3ba" (UID: "501a2c31-fb1e-47b9-acf7-aff6367ef3ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:35.568366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.568265    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-kube-api-access-ztrjh" pod "501a2c31-fb1e-47b9-acf7-aff6367ef3ba" (UID: "501a2c31-fb1e-47b9-acf7-aff6367ef3ba"). InnerVolumeSpecName "kube-api-access-ztrjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:35.575762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.575731    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:35.575973 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.575957    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztrjh\" (UniqueName: \"kubernetes.io/projected/501a2c31-fb1e-47b9-acf7-aff6367ef3ba-kube-api-access-ztrjh\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:35.673920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.673891    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:03:35.822963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.809523    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d984bb8-3ed8-4790-827a-4292af0cfcd2-kube-api-access-74vwz\" (UniqueName: \"kubernetes.io/projected/8d984bb8-3ed8-4790-827a-4292af0cfcd2-kube-api-access-74vwz\") pod \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\" (UID: \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\") "
Oct 30 22:03:35.862679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.862639    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d984bb8-3ed8-4790-827a-4292af0cfcd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d984bb8-3ed8-4790-827a-4292af0cfcd2-plugins-config-volume\") pod \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\" (UID: \"8d984bb8-3ed8-4790-827a-4292af0cfcd2\") "
Oct 30 22:03:35.935882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.935840    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d984bb8-3ed8-4790-827a-4292af0cfcd2-plugins-config-volume" pod "8d984bb8-3ed8-4790-827a-4292af0cfcd2" (UID: "8d984bb8-3ed8-4790-827a-4292af0cfcd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:35.980680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:35.980642    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d984bb8-3ed8-4790-827a-4292af0cfcd2-kube-api-access-74vwz" pod "8d984bb8-3ed8-4790-827a-4292af0cfcd2" (UID: "8d984bb8-3ed8-4790-827a-4292af0cfcd2"). InnerVolumeSpecName "kube-api-access-74vwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:36.023402 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.023278    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vwz\" (UniqueName: \"kubernetes.io/projected/8d984bb8-3ed8-4790-827a-4292af0cfcd2-kube-api-access-74vwz\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:36.023402 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.023317    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d984bb8-3ed8-4790-827a-4292af0cfcd2-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:36.172180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.172135    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc" event={"ID":"8d984bb8-3ed8-4790-827a-4292af0cfcd2","Type":"ContainerDied","Data":"25f5d03e40d815d1cd3a3d1fff0ab7ba7c66760346549afd4320f5138fedc651"}
Oct 30 22:03:36.172391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.172268    4282 scope.go:122] "RemoveContainer" containerID="11f7ae5ed6c8d52371cd85c53a71551950d4d8bd798268ec298304281a07db11"
Oct 30 22:03:36.172539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.172520    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"
Oct 30 22:03:36.185158 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.185122    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"
Oct 30 22:03:36.187205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.187181    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48" event={"ID":"501a2c31-fb1e-47b9-acf7-aff6367ef3ba","Type":"ContainerDied","Data":"c5a9a1c558a34c142392ef9927b377ff058b2f9b77ab06341b5eee140e06a1d7"}
Oct 30 22:03:36.252280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.252238    4282 scope.go:122] "RemoveContainer" containerID="ae96f8ae3b174af0137035b97c659a45c4bbbbdf71d538fbef3df5064ac6d0ad"
Oct 30 22:03:36.453301 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.453217    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"]
Oct 30 22:03:36.459330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.459296    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-90b6f5-7bf46c7d9-fl5pc"]
Oct 30 22:03:36.464697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.464651    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"]
Oct 30 22:03:36.469860 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:36.469835    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-90b6f5-8548cf45bc-hcd48"]
Oct 30 22:03:37.580501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:37.576778    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="501a2c31-fb1e-47b9-acf7-aff6367ef3ba" path="/var/lib/kubelet/pods/501a2c31-fb1e-47b9-acf7-aff6367ef3ba/volumes"
Oct 30 22:03:37.598609 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:37.583976    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d984bb8-3ed8-4790-827a-4292af0cfcd2" path="/var/lib/kubelet/pods/8d984bb8-3ed8-4790-827a-4292af0cfcd2/volumes"
Oct 30 22:03:38.335048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.316626    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:03:38.339323 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.339290    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"]
Oct 30 22:03:38.339763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.339729    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy" containerID="cri-o://cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b" gracePeriod=30
Oct 30 22:03:38.341825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.341800    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded"
Oct 30 22:03:38.597618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.590808    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:38.597618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.592708    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"]
Oct 30 22:03:38.597618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.593984    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8d984bb8-3ed8-4790-827a-4292af0cfcd2" containerName="payload-processing"
Oct 30 22:03:38.597618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.594007    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d984bb8-3ed8-4790-827a-4292af0cfcd2" containerName="payload-processing"
Oct 30 22:03:38.597618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.594067    4282 cpu_manager.go:414] "containerMap: removing container" podUID="501a2c31-fb1e-47b9-acf7-aff6367ef3ba" containerName="payload-pre-processing"
Oct 30 22:03:38.597618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.594076    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="501a2c31-fb1e-47b9-acf7-aff6367ef3ba" containerName="payload-pre-processing"
Oct 30 22:03:38.597618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.594212    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="501a2c31-fb1e-47b9-acf7-aff6367ef3ba" containerName="payload-pre-processing"
Oct 30 22:03:38.597618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.594236    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d984bb8-3ed8-4790-827a-4292af0cfcd2" containerName="payload-processing"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.741091    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"]
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.741252    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744007    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744105    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjfz\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-kube-api-access-7rjfz\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744155    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744182    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744225    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744262    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744335    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c2a15e-52fa-4af8-837f-b1a5b0430643-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744528    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.745492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.744638    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846410    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjfz\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-kube-api-access-7rjfz\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846456    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846519    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846556    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846586    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846650    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c2a15e-52fa-4af8-837f-b1a5b0430643-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846762    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846838    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.846944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846924    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.847100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.846989    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.847100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.847034    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.847212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.847112    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.847212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.847190    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.847410 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.847392    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c2a15e-52fa-4af8-837f-b1a5b0430643-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.849414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.849365    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.849549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.849530    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.859309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.859285    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjfz\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-kube-api-access-7rjfz\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:38.859671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:38.859641    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:39.091494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:39.086054    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:39.423213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:39.423188    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"]
Oct 30 22:03:39.423904 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:03:39.423876    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c2a15e_52fa_4af8_837f_b1a5b0430643.slice/crio-caadabe7b241a8993f2f1504296902508260eca6ef9ee23b01e8e9ecc33eff91 WatchSource:0}: Error finding container caadabe7b241a8993f2f1504296902508260eca6ef9ee23b01e8e9ecc33eff91: Status 404 returned error can't find the container with id caadabe7b241a8993f2f1504296902508260eca6ef9ee23b01e8e9ecc33eff91
Oct 30 22:03:39.426855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:39.426818    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:39.426965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:39.426891    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:39.426965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:39.426924    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:03:40.386805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:40.386768    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" event={"ID":"66c2a15e-52fa-4af8-837f-b1a5b0430643","Type":"ContainerStarted","Data":"a49bb9e396c60fd0818305841bfe6840372cf2a511680e6fd625316407457d7d"}
Oct 30 22:03:40.387310 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:40.386825    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" event={"ID":"66c2a15e-52fa-4af8-837f-b1a5b0430643","Type":"ContainerStarted","Data":"caadabe7b241a8993f2f1504296902508260eca6ef9ee23b01e8e9ecc33eff91"}
Oct 30 22:03:40.424417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:40.424360    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" podStartSLOduration=2.424340967 podStartE2EDuration="2.424340967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:40.422177572 +0000 UTC m=+1439.080273794" watchObservedRunningTime="2026-10-30 22:03:40.424340967 +0000 UTC m=+1439.082437191"
Oct 30 22:03:41.087548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:41.087513    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:41.088890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:41.088861    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:03:41.089022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:41.088922    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:03:41.986028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:41.986002    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:03:42.087705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:42.087671    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:03:42.087855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:42.087733    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:03:42.301547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:42.299703    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:03:43.095596 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.095558    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:03:43.096023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.095626    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:03:43.706225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.705938    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:03:43.943586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943556    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-podinfo\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.943777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943616    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-kube-api-access-j6cm2\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-kube-api-access-j6cm2\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.943777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943666    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-socket\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.943910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943789    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bb00c7-522d-4e8a-baf0-95897d40452c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bb00c7-522d-4e8a-baf0-95897d40452c-istiod-ca-cert\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.943910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943813    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-credential-socket\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.943910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943843    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-envoy\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.943910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943874    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-data\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.944141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943969    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-token\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-token\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.944141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.943997    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-certs\") pod \"b0bb00c7-522d-4e8a-baf0-95897d40452c\" (UID: \"b0bb00c7-522d-4e8a-baf0-95897d40452c\") "
Oct 30 22:03:43.944141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944057    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-socket" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:43.944301 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944194    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bb00c7-522d-4e8a-baf0-95897d40452c-istiod-ca-cert" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:43.944301 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944220    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-data" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:43.944610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944320    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-credential-socket" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:43.944610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944385    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-certs" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:43.944953 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944928    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0bb00c7-522d-4e8a-baf0-95897d40452c-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:43.945025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944959    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:43.945025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944974    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:43.945025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944986    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:43.945025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.944999    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:43.946105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.946083    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-podinfo" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:43.946105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.946091    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-kube-api-access-j6cm2" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "kube-api-access-j6cm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:43.946606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.946580    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-envoy" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:43.946676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:43.946586    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-token" pod "b0bb00c7-522d-4e8a-baf0-95897d40452c" (UID: "b0bb00c7-522d-4e8a-baf0-95897d40452c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:44.045873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.045842    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.045873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.045870    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.045873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.045880    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0bb00c7-522d-4e8a-baf0-95897d40452c-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.046084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.045890    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6cm2\" (UniqueName: \"kubernetes.io/projected/b0bb00c7-522d-4e8a-baf0-95897d40452c-kube-api-access-j6cm2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:44.087523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.087461    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:03:44.087675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.087557    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:03:44.158495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.151482    4282 scope.go:122] "RemoveContainer" containerID="cbc01f4c5f2fe1d00bb6e9147b790e0da365c91f0248acb0899db4b85e527452"
Oct 30 22:03:44.419005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.418975    4282 generic.go:356] "Generic (PLEG): container finished" podID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerID="cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b" exitCode=0
Oct 30 22:03:44.419180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.419016    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" event={"ID":"b0bb00c7-522d-4e8a-baf0-95897d40452c","Type":"ContainerDied","Data":"cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b"}
Oct 30 22:03:44.419180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.419026    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"
Oct 30 22:03:44.419180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.419057    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk" event={"ID":"b0bb00c7-522d-4e8a-baf0-95897d40452c","Type":"ContainerDied","Data":"e0f8565e359949ce3e6fc7146f216f0190a8825f6d24247953c78556e4af2865"}
Oct 30 22:03:44.419180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.419078    4282 scope.go:122] "RemoveContainer" containerID="cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b"
Oct 30 22:03:44.431909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.429814    4282 scope.go:122] "RemoveContainer" containerID="0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d"
Oct 30 22:03:44.509390 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.507538    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"]
Oct 30 22:03:44.509390 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.509296    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-90b6f5-openshift-default-6bdf4cd857-5vrlk"]
Oct 30 22:03:44.538458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.538342    4282 scope.go:122] "RemoveContainer" containerID="cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b"
Oct 30 22:03:44.539159 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:44.539017    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b\": container with ID starting with cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b not found: ID does not exist" containerID="cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b"
Oct 30 22:03:44.539159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.539053    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b"} err="failed to get container status \"cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b\": rpc error: code = NotFound desc = could not find container \"cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b\": container with ID starting with cb5707174602049f94ef5262fb8a23eb3fca8507aebf34cd6a4c15fe2d64c75b not found: ID does not exist"
Oct 30 22:03:44.539159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.539080    4282 scope.go:122] "RemoveContainer" containerID="0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d"
Oct 30 22:03:44.539802 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:03:44.539738    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d\": container with ID starting with 0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d not found: ID does not exist" containerID="0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d"
Oct 30 22:03:44.539802 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:44.539769    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d"} err="failed to get container status \"0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d\": rpc error: code = NotFound desc = could not find container \"0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d\": container with ID starting with 0fc3d65d0faa2832a6e6fa169f255b772859e142e32bc37fa5eed670e764091d not found: ID does not exist"
Oct 30 22:03:45.087277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:45.087238    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:03:45.087439 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:45.087294    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:03:45.476720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:45.476688    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" path="/var/lib/kubelet/pods/b0bb00c7-522d-4e8a-baf0-95897d40452c/volumes"
Oct 30 22:03:46.087336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:46.087301    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Oct 30 22:03:46.087532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:46.087360    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Oct 30 22:03:47.256499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:47.243104    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:47.256499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:47.244614    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:47.256499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:47.248012    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:03:54.632566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.632533    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"]
Oct 30 22:03:54.633156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.633140    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy"
Oct 30 22:03:54.633156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.633156    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy"
Oct 30 22:03:54.633313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.633184    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy"
Oct 30 22:03:54.633313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.633190    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy"
Oct 30 22:03:54.633313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.633275    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy"
Oct 30 22:03:54.633313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.633289    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bb00c7-522d-4e8a-baf0-95897d40452c" containerName="istio-proxy"
Oct 30 22:03:54.639230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.639212    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.642966 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.642923    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-5d81a2\"/\"praxis-route-5d81a2-kserve-self-signed-certs\""
Oct 30 22:03:54.643637 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.643617    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-5d81a2\"/\"default-dockercfg-85hcs\""
Oct 30 22:03:54.646036 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.646017    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-5d81a2\"/\"openshift-service-ca.crt\""
Oct 30 22:03:54.649824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.649806    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-5d81a2\"/\"kube-root-ca.crt\""
Oct 30 22:03:54.653581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.653557    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"]
Oct 30 22:03:54.712084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.712051    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-home\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.712236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.712101    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-tmp-dir\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.712310 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.712257    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7l2v\" (UniqueName: \"kubernetes.io/projected/97475a44-715c-4ff6-8f75-a54fc074be6a-kube-api-access-r7l2v\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.712364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.712349    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-dshm\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.712402 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.712389    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-model-cache\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.712746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.712568    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97475a44-715c-4ff6-8f75-a54fc074be6a-tls-certs\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814218    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-dshm\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814276    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-model-cache\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814318    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97475a44-715c-4ff6-8f75-a54fc074be6a-tls-certs\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814384    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-home\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814412    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-tmp-dir\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814608    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l2v\" (UniqueName: \"kubernetes.io/projected/97475a44-715c-4ff6-8f75-a54fc074be6a-kube-api-access-r7l2v\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814764 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814737    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-model-cache\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814776    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-home\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.814888 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.814822    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-tmp-dir\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.816677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.816659    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-dshm\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.816999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.816982    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97475a44-715c-4ff6-8f75-a54fc074be6a-tls-certs\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.826042 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.826016    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7l2v\" (UniqueName: \"kubernetes.io/projected/97475a44-715c-4ff6-8f75-a54fc074be6a-kube-api-access-r7l2v\") pod \"praxis-route-5d81a2-kserve-bbf66979f-zk8vw\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") " pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:54.979494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:54.972766    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:55.335572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:55.335545    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"]
Oct 30 22:03:55.336961 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:03:55.336936    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97475a44_715c_4ff6_8f75_a54fc074be6a.slice/crio-f3f247c05bfeed2636267eeb2944fde87ab116a07e0c168f3cad674e8e155602 WatchSource:0}: Error finding container f3f247c05bfeed2636267eeb2944fde87ab116a07e0c168f3cad674e8e155602: Status 404 returned error can't find the container with id f3f247c05bfeed2636267eeb2944fde87ab116a07e0c168f3cad674e8e155602
Oct 30 22:03:55.491332 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:55.491297    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" event={"ID":"97475a44-715c-4ff6-8f75-a54fc074be6a","Type":"ContainerStarted","Data":"6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c"}
Oct 30 22:03:55.491545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:55.491338    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" event={"ID":"97475a44-715c-4ff6-8f75-a54fc074be6a","Type":"ContainerStarted","Data":"f3f247c05bfeed2636267eeb2944fde87ab116a07e0c168f3cad674e8e155602"}
Oct 30 22:03:55.513005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:55.512954    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podStartSLOduration=1.512941494 podStartE2EDuration="1.512941494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:55.508975662 +0000 UTC m=+1454.167071884" watchObservedRunningTime="2026-10-30 22:03:55.512941494 +0000 UTC m=+1454.171037717"
Oct 30 22:03:55.974440 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:55.974410    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:03:59.298640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:59.298592    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:03:59.355623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:59.355577    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.486898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:59.486854    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.601659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:59.601556    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:59.974657 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:03:59.974614    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:04:00.143072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.142597    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.227553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.215714    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.301631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.301577    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:00.303269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.302347    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:04:00.381590 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.379745    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.381590 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.380312    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:04:00.530601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.528722    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.530601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.529250    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:04:00.530601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.530578    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.723512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.711682    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.723512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.712272    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:04:00.934668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.916605    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:00.977593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.977548    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:00.992391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.978715    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:04:01.137033 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:00.920566    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:01.532519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:01.530540    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.548509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:01.547562    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.571713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:01.571679    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:04:01.635557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:01.605894    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:01.635557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:01.617990    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:04:01.726825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:01.723726    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:01.982777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:01.982733    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:04:02.615505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:02.572760    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:02.668861 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:02.668813    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:04:02.760557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:02.695225    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:02.899067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:02.898947    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:04:03.099688 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.099649    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.119740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.119703    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.279012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.278978    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:04:03.587541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.582273    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.859915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.859724    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:04:03.866233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.866206    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:04:03.882361 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.869420    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:04:03.902735 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.871134    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:04:03.903805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.871712    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:03.932930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.932907    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:04:03.933557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:03.933535    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:04:04.650255 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:04:04.650204    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:04:04.976675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:04.976639    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:04:04.989638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:04.989613    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:04:05.366807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:05.366723    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:04:11.849043 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:11.848104    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": read tcp 10.133.0.2:38008->10.133.0.148:15021: read: connection reset by peer" start-of-body=
Oct 30 22:04:11.849043 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:11.848572    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": read tcp 10.133.0.2:38008->10.133.0.148:15021: read: connection reset by peer"
Oct 30 22:04:12.360509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:12.360229    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:04:12.420405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:12.418441    4282 generic.go:356] "Generic (PLEG): container finished" podID="822606a9-93cc-468f-99d6-6a040df39808" containerID="37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362" exitCode=0
Oct 30 22:04:12.420405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:12.418581    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" event={"ID":"822606a9-93cc-468f-99d6-6a040df39808","Type":"ContainerDied","Data":"37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362"}
Oct 30 22:04:12.420405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:12.419112    4282 scope.go:122] "RemoveContainer" containerID="37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362"
Oct 30 22:04:12.422522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:12.422071    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:12.422522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:12.422164    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:12.422522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:12.422209    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:13.427195 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:13.427160    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" event={"ID":"822606a9-93cc-468f-99d6-6a040df39808","Type":"ContainerStarted","Data":"a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0"}
Oct 30 22:04:13.842129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:13.841634    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:04:13.842519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:13.842018    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:04:13.842683 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:13.842661    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 22:04:14.641489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:14.637994    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.676818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:14.676749    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 22:04:14.883494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:14.857796    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.883494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:14.867763    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:04:14.883494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:14.867832    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 22:04:14.908309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:14.907580    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:04:14.908309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:14.907985    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.018741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.018686    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:04:15.347501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.345564    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.547705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.547659    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.638675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.638461    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.650065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.650032    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:04:15.871708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.871671    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:04:15.872024 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.871979    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 22:04:15.927171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.927127    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:15.927919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.927898    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:04:15.950563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.950526    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.057172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.963816    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.057759 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:15.997605    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.057970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.034535    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.148879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.073752    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:04:16.148879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.073824    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:04:16.148879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.073863    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:04:16.346335 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.346236    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": context deadline exceeded"
Oct 30 22:04:16.347962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.347936    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:04:16.561599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.561559    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.563042 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.563017    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:04:16.653586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.646800    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:16.847770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.847738    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:04:16.848147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.848095    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 22:04:16.929658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:16.929609    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.067618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.067567    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.068534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.068503    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.070408 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.070380    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.357602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.347266    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.587811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.587763    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.652669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.652562    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:17.812438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.812123    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:04:17.813108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.813085    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:04:17.813710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.813691    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:04:17.815189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.815160    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:04:17.815706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.815687    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:04:17.830828 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.830806    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:04:17.837748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.837697    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:04:17.841399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.841378    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:04:17.841594 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:17.841572    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 22:04:18.857490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:18.841869    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:04:18.857490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:18.841930    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 22:04:19.841586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:19.841551    4282 patch_prober.go:29] interesting pod/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Oct 30 22:04:19.841766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:19.841598    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Oct 30 22:04:20.846716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:20.846685    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:04:20.847194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:20.846998    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:04:20.847700 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:20.847682    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:04:23.403142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.403074    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"]
Oct 30 22:04:23.407604 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.407588    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.421168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.421144    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"]
Oct 30 22:04:23.477730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.477694    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.477915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.477761    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.477915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.477866    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.478029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.477999    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.478068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.478028    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.478122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.478070    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.478209 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.478191    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.478307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.478292    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.478364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.478351    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc78p\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-kube-api-access-mc78p\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579234    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579302    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579344    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579444    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579662 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579517    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579662 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579589    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579667    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579730    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579763    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.579879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579814    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc78p\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-kube-api-access-mc78p\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.580019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.579996    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.580093 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.580022    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.580155 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.580119    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.580362 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.580341    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.582441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.582412    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.582635 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.582618    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.591188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.591167    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc78p\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-kube-api-access-mc78p\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.591391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.591370    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:23.729376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:23.729336    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:24.029143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:24.029120    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"]
Oct 30 22:04:24.031200 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:04:24.031174    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75bd9566_2ab1_4aeb_b273_f6ad727bd1d4.slice/crio-59dd30041019bcbd86628a4444f46abd407d817f5e65f22357648be51588b6bd WatchSource:0}: Error finding container 59dd30041019bcbd86628a4444f46abd407d817f5e65f22357648be51588b6bd: Status 404 returned error can't find the container with id 59dd30041019bcbd86628a4444f46abd407d817f5e65f22357648be51588b6bd
Oct 30 22:04:24.033830 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:24.033795    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:24.033909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:24.033863    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:24.033909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:24.033893    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:24.853037 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:24.853000    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" event={"ID":"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4","Type":"ContainerStarted","Data":"a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392"}
Oct 30 22:04:24.853685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:24.853660    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" event={"ID":"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4","Type":"ContainerStarted","Data":"59dd30041019bcbd86628a4444f46abd407d817f5e65f22357648be51588b6bd"}
Oct 30 22:04:25.729844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.729817    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:25.731983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.731954    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:04:25.733023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.732878    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:04:25.777977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.777911    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podStartSLOduration=2.777889463 podStartE2EDuration="2.777889463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:24.883868129 +0000 UTC m=+1483.541964353" watchObservedRunningTime="2026-10-30 22:04:25.777889463 +0000 UTC m=+1484.435985686"
Oct 30 22:04:25.781754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.781727    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"]
Oct 30 22:04:25.789527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.787117    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.793234 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.793029    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"]
Oct 30 22:04:25.809904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.809865    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.810044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.810022    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b65g\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-kube-api-access-7b65g\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.810132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.810116    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.810456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.810435    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.810633 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.810613    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.810684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.810670    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.810729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.810703    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159aeca-66aa-4b6b-b7cb-720375678e70-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.810899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.810881    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.810950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.810936    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.912798 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.912759    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b65g\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-kube-api-access-7b65g\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.912856    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913057    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913131    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913157    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913185    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159aeca-66aa-4b6b-b7cb-720375678e70-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913290    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913323    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913664 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913347    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913664 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913377    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913788    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.913861 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.913827    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.914118 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.914089    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.914252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.914236    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159aeca-66aa-4b6b-b7cb-720375678e70-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.916115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.916094    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.916249 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.916231    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.929635 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.929613    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:25.929725 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:25.929637    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b65g\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-kube-api-access-7b65g\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:26.113809 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.113717    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:26.503904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.501793    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"]
Oct 30 22:04:26.511325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.511278    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:26.511450 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.511364    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:26.511450 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.511394    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:26.734494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.734046    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:04:26.734494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.734102    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:04:26.895540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.894510    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" event={"ID":"d159aeca-66aa-4b6b-b7cb-720375678e70","Type":"ContainerStarted","Data":"4fe9b9c4f9e0604fdd13e6f519af445ec1a76ec025f46f076c406e9b6fbad25b"}
Oct 30 22:04:26.895540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.894575    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" event={"ID":"d159aeca-66aa-4b6b-b7cb-720375678e70","Type":"ContainerStarted","Data":"ae9a55517dc2234870c5fa119c4fea2013a8b4b1f7d1be3cc793a1091b3d3eb0"}
Oct 30 22:04:26.916745 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:26.916693    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podStartSLOduration=1.91667847 podStartE2EDuration="1.91667847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:26.916362482 +0000 UTC m=+1485.574458705" watchObservedRunningTime="2026-10-30 22:04:26.91667847 +0000 UTC m=+1485.574774694"
Oct 30 22:04:27.113939 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.113900    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:27.115060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.115035    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:27.115213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.115080    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:27.629553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.629520    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"]
Oct 30 22:04:27.633853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.633835    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.645001 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.644979    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5d81a2-openshift-default-dockercfg-jlhhw\""
Oct 30 22:04:27.731527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.731170    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:04:27.731527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.731225    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:04:27.735342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.735313    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"]
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.814383    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-envoy\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.816314    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.816421    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-credential-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.816616    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-data\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.816738    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-certs\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.816767    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-podinfo\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.816795    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqn4\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-kube-api-access-7qqn4\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.816843    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istiod-ca-cert\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.824881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.816907    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-token\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.919963    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-data\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.920167    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-certs\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.920308    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-podinfo\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.920345    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqn4\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-kube-api-access-7qqn4\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.920396    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istiod-ca-cert\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.920518    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-token\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.920591    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-envoy\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.920684    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.920746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.920727    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-credential-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.921225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.921200    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-credential-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.921342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.921306    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-data\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.921516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.921449    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-certs\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.921773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.921716    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-socket\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.921959 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.921933    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istiod-ca-cert\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.923903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.923879    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-podinfo\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.925023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.925005    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-envoy\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.932304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.932277    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-token\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.932438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.932420    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqn4\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-kube-api-access-7qqn4\") pod \"e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") " pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:27.948652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:27.948631    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:28.119884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.119680    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:28.119884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.119736    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:28.407542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.405627    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"]
Oct 30 22:04:28.508496 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:04:28.507987    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ebeb51_11ed_4109_bf1f_5281e08a6445.slice/crio-e14b8b3226d0dafec407adb92d58d331d5a23e0842191d069e96f832bb6235d9 WatchSource:0}: Error finding container e14b8b3226d0dafec407adb92d58d331d5a23e0842191d069e96f832bb6235d9: Status 404 returned error can't find the container with id e14b8b3226d0dafec407adb92d58d331d5a23e0842191d069e96f832bb6235d9
Oct 30 22:04:28.520781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.511813    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:28.520781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.511918    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:28.520781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.511980    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:28.730411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.730375    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:04:28.730604 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.730433    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:04:28.936386 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.936345    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" event={"ID":"b3ebeb51-11ed-4109-bf1f-5281e08a6445","Type":"ContainerStarted","Data":"94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e"}
Oct 30 22:04:28.936796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.936456    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" event={"ID":"b3ebeb51-11ed-4109-bf1f-5281e08a6445","Type":"ContainerStarted","Data":"e14b8b3226d0dafec407adb92d58d331d5a23e0842191d069e96f832bb6235d9"}
Oct 30 22:04:28.949356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.949332    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:28.971494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:28.971146    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podStartSLOduration=1.971127471 podStartE2EDuration="1.971127471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:28.969786538 +0000 UTC m=+1487.627882784" watchObservedRunningTime="2026-10-30 22:04:28.971127471 +0000 UTC m=+1487.629223695"
Oct 30 22:04:29.116549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:29.115892    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:29.116549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:29.115963    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:29.740224 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:29.740184    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:04:29.740416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:29.740262    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:04:30.008563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.004641    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:04:30.008563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.004707    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 22:04:30.227535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.224026    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:30.227535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.224097    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:30.735930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.731032    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:04:30.735930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.731089    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:04:30.823706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.823663    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"]
Oct 30 22:04:30.829206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.829161    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:30.837904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.837806    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e5a72d-e1d9-4216-af8d-1131d1afe450-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx\" (UID: \"58e5a72d-e1d9-4216-af8d-1131d1afe450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:30.838057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.837959    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqw9b\" (UniqueName: \"kubernetes.io/projected/58e5a72d-e1d9-4216-af8d-1131d1afe450-kube-api-access-hqw9b\") pod \"payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx\" (UID: \"58e5a72d-e1d9-4216-af8d-1131d1afe450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:30.839775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.839007    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-18ba2f\""
Oct 30 22:04:30.846568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.843434    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"]
Oct 30 22:04:30.939856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.939816    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e5a72d-e1d9-4216-af8d-1131d1afe450-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx\" (UID: \"58e5a72d-e1d9-4216-af8d-1131d1afe450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:30.940226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.940200    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqw9b\" (UniqueName: \"kubernetes.io/projected/58e5a72d-e1d9-4216-af8d-1131d1afe450-kube-api-access-hqw9b\") pod \"payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx\" (UID: \"58e5a72d-e1d9-4216-af8d-1131d1afe450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:30.941082 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.941052    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e5a72d-e1d9-4216-af8d-1131d1afe450-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx\" (UID: \"58e5a72d-e1d9-4216-af8d-1131d1afe450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:30.949664 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.949640    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:04:30.949839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:30.949814    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 22:04:31.022676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:31.022589    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqw9b\" (UniqueName: \"kubernetes.io/projected/58e5a72d-e1d9-4216-af8d-1131d1afe450-kube-api-access-hqw9b\") pod \"payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx\" (UID: \"58e5a72d-e1d9-4216-af8d-1131d1afe450\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:31.139526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:31.139437    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:31.139791 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:31.139761    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:31.214971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:31.214873    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:31.943092 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:31.943046    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Oct 30 22:04:31.943288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:31.943113    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Oct 30 22:04:31.944653 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:04:31.944620    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e5a72d_e1d9_4216_af8d_1131d1afe450.slice/crio-3fb989a37c4a70c07ecaa71012d7f1e88350f199415c52501393b4642e3bf7be WatchSource:0}: Error finding container 3fb989a37c4a70c07ecaa71012d7f1e88350f199415c52501393b4642e3bf7be: Status 404 returned error can't find the container with id 3fb989a37c4a70c07ecaa71012d7f1e88350f199415c52501393b4642e3bf7be
Oct 30 22:04:31.952622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:31.952196    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:04:31.952622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:31.952234    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 22:04:32.034220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.034190    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"]
Oct 30 22:04:32.129739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.127574    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"]
Oct 30 22:04:32.137103 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.136688    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:32.137103 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.136739    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:32.144436 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.144154    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"]
Oct 30 22:04:32.144436 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.144293    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.147727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.147691    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ecfbd6-openshift-default-dockercfg-b5wnh\""
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.392673    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-socket\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.392745    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-credential-socket\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.392882    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istiod-ca-cert\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.392938    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-token\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.392999    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-certs\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.393036    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwc7g\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-kube-api-access-cwc7g\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.393087    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-data\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.393124    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-podinfo\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.403081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.393211    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-envoy\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.453846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.453805    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx" event={"ID":"58e5a72d-e1d9-4216-af8d-1131d1afe450","Type":"ContainerStarted","Data":"3fb989a37c4a70c07ecaa71012d7f1e88350f199415c52501393b4642e3bf7be"}
Oct 30 22:04:32.498492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.496804    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istiod-ca-cert\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.498492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.496900    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-token\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.498492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.496978    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-certs\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.498492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.497033    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwc7g\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-kube-api-access-cwc7g\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.504639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.504601    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-data\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.504930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.504906    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-podinfo\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.505208 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.505175    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-envoy\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.505539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.505514    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-socket\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.505747 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.505715    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-credential-socket\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.505984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.505957    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istiod-ca-cert\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.512193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.512156    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-certs\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.512724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.512696    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-credential-socket\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.523248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.523211    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-data\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.530257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.527633    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwc7g\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-kube-api-access-cwc7g\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.546006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.540966    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-token\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.546006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.542205    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-socket\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.562971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.561404    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-podinfo\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.587106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.587071    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-envoy\") pod \"e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") " pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.625060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.625017    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:32.996213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.991938    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:04:32.996213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:32.991995    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 22:04:33.129323 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.128816    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:33.129323 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.128917    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:33.638130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.638053    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx" event={"ID":"58e5a72d-e1d9-4216-af8d-1131d1afe450","Type":"ContainerStarted","Data":"90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8"}
Oct 30 22:04:33.638339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.638327    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:33.744415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.743746    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx" podStartSLOduration=3.743728761 podStartE2EDuration="3.743728761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:33.693741512 +0000 UTC m=+1492.351837737" watchObservedRunningTime="2026-10-30 22:04:33.743728761 +0000 UTC m=+1492.401824984"
Oct 30 22:04:33.750210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.750179    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"]
Oct 30 22:04:33.876710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.875353    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:33.876710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.875412    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:33.948538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.930538    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:33.948538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.934684    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:33.948538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.934870    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:33.969523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.951573    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:04:33.969523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:33.959367    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 22:04:34.107627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.102789    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:34.107627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.104493    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:34.141542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.138823    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:04:34.141542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.139200    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:34.141542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.139245    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:34.658559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.654293    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" event={"ID":"f3b514a9-b9c4-4705-a789-a7acaba6a27f","Type":"ContainerStarted","Data":"df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d"}
Oct 30 22:04:34.658559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.654350    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" event={"ID":"f3b514a9-b9c4-4705-a789-a7acaba6a27f","Type":"ContainerStarted","Data":"c776b3de776771bdbc6bb886ffed5540e623fde9689bf941f15b26083ab5916e"}
Oct 30 22:04:34.661640 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:04:34.661583    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:04:34.961495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.957607    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:04:34.961495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:34.957665    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 22:04:35.115495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.114873    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 22:04:35.115495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.114932    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 22:04:35.630984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.630219    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:35.637388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.637360    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:35.700652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.700590    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" podStartSLOduration=3.700570668 podStartE2EDuration="3.700570668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:34.696441929 +0000 UTC m=+1493.354538155" watchObservedRunningTime="2026-10-30 22:04:35.700570668 +0000 UTC m=+1494.358666893"
Oct 30 22:04:35.757500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.751695    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:35.785727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.783807    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:04:35.794828 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.794793    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"]
Oct 30 22:04:35.801843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.801815    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:35.819890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.819777    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cca1c7\""
Oct 30 22:04:35.824627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.820589    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"]
Oct 30 22:04:35.836249 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.835041    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cca1c7-dockercfg-6bdpz\""
Oct 30 22:04:35.849648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.848235    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhz2\" (UniqueName: \"kubernetes.io/projected/9fd149ee-426a-4670-b647-02f50aef6219-kube-api-access-hqhz2\") pod \"payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w\" (UID: \"9fd149ee-426a-4670-b647-02f50aef6219\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:35.849648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.848369    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd149ee-426a-4670-b647-02f50aef6219-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w\" (UID: \"9fd149ee-426a-4670-b647-02f50aef6219\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:35.949248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.949018    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:04:35.949248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:35.949105    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 22:04:36.016499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.005295    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhz2\" (UniqueName: \"kubernetes.io/projected/9fd149ee-426a-4670-b647-02f50aef6219-kube-api-access-hqhz2\") pod \"payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w\" (UID: \"9fd149ee-426a-4670-b647-02f50aef6219\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:36.016499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.005372    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd149ee-426a-4670-b647-02f50aef6219-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w\" (UID: \"9fd149ee-426a-4670-b647-02f50aef6219\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:36.038491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.018674    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd149ee-426a-4670-b647-02f50aef6219-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w\" (UID: \"9fd149ee-426a-4670-b647-02f50aef6219\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:36.195491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.193864    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhz2\" (UniqueName: \"kubernetes.io/projected/9fd149ee-426a-4670-b647-02f50aef6219-kube-api-access-hqhz2\") pod \"payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w\" (UID: \"9fd149ee-426a-4670-b647-02f50aef6219\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:36.218542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.214559    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"]
Oct 30 22:04:36.262574 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.259427    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.306294 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.300367    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-18ba2f-serving-cert\""
Oct 30 22:04:36.329520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.328185    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"]
Oct 30 22:04:36.419487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.418590    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.419487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.418672    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.419487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.418698    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zklp\" (UniqueName: \"kubernetes.io/projected/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-kube-api-access-9zklp\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.419487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.418810    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.442084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.442044    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:36.551513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.543539    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.551513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.543662    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.551513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.543694    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zklp\" (UniqueName: \"kubernetes.io/projected/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-kube-api-access-9zklp\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.583957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.583897    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.584967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.584290    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.588636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.588570    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.675508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.671555    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.769642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.753721    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zklp\" (UniqueName: \"kubernetes.io/projected/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-kube-api-access-9zklp\") pod \"maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:36.949882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.949364    4282 patch_prober.go:29] interesting pod/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 22:04:36.949882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:36.949427    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 22:04:37.015638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:37.014604    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:37.015970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:37.015928    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:37.020306 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:37.017041    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:04:37.047790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:37.047761    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:37.096988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:37.096954    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"]
Oct 30 22:04:37.398620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:37.398597    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"]
Oct 30 22:04:37.974685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:37.974621    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"]
Oct 30 22:04:38.029530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.021668    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"]
Oct 30 22:04:38.029530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.021785    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:38.029530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.023169    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:38.048516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.030377    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:38.048516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.034732    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5313f0\""
Oct 30 22:04:38.066801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.066771    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:04:38.089518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.072172    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca26915d-a5a8-4111-aca6-54ee76f8cad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb\" (UID: \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:38.089518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.072270    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdm2\" (UniqueName: \"kubernetes.io/projected/ca26915d-a5a8-4111-aca6-54ee76f8cad9-kube-api-access-rbdm2\") pod \"payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb\" (UID: \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:38.241571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.209452    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w" event={"ID":"9fd149ee-426a-4670-b647-02f50aef6219","Type":"ContainerStarted","Data":"278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414"}
Oct 30 22:04:38.241571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.231697    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:38.241571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.231730    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w" event={"ID":"9fd149ee-426a-4670-b647-02f50aef6219","Type":"ContainerStarted","Data":"43abb7a8560b0d5ce7c8e6ca61927703d302a8abdc2411dcdca7aca10f7de917"}
Oct 30 22:04:38.301509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.297558    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca26915d-a5a8-4111-aca6-54ee76f8cad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb\" (UID: \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:38.301509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.297675    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdm2\" (UniqueName: \"kubernetes.io/projected/ca26915d-a5a8-4111-aca6-54ee76f8cad9-kube-api-access-rbdm2\") pod \"payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb\" (UID: \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:38.301509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.299378    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca26915d-a5a8-4111-aca6-54ee76f8cad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb\" (UID: \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:38.409655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.409590    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7" event={"ID":"fdb43687-6eea-4cf8-98ac-1810e4d72e9d","Type":"ContainerStarted","Data":"5a7e97fe50602360fd22eda82ea0d966ed694ce157e1f03b4dce8c8a88048f60"}
Oct 30 22:04:38.704500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.676711    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w" podStartSLOduration=3.676685389 podStartE2EDuration="3.676685389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:38.368976492 +0000 UTC m=+1497.027072719" watchObservedRunningTime="2026-10-30 22:04:38.676685389 +0000 UTC m=+1497.334781612"
Oct 30 22:04:38.799495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.735951    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdm2\" (UniqueName: \"kubernetes.io/projected/ca26915d-a5a8-4111-aca6-54ee76f8cad9-kube-api-access-rbdm2\") pod \"payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb\" (UID: \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:38.822512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.818716    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.857542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.838117    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:38.975444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:38.975413    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:39.089333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.089302    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:04:39.333182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.332594    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:04:39.577259 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.577182    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7" podStartSLOduration=2.53586139 podStartE2EDuration="3.577160384s" totalImagesPullingTime="1.041298994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:39.558260736 +0000 UTC m=+1498.216356959" watchObservedRunningTime="2026-10-30 22:04:39.577160384 +0000 UTC m=+1498.235256606"
Oct 30 22:04:39.675424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.675385    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7" event={"ID":"fdb43687-6eea-4cf8-98ac-1810e4d72e9d","Type":"ContainerStarted","Data":"9591ffb84a27ae36758e5d6380794238c41a5fd9c09b6c786d3f3a916da35f14"}
Oct 30 22:04:39.712507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.709663    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:39.712507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.709764    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:39.712507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.710157    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:04:39.837630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.835696    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.837630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.836268    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:04:39.886560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.838894    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:39.886560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.839356    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:04:39.886560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:39.840081    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.102076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.101849    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.135068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.113666    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 30 22:04:40.162137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.162093    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:04:40.374459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.354289    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.374459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.357862    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:04:40.522104 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.522062    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.725495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.712447    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:40.725495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.713658    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:40.771512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.734701    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:04:40.771512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:40.734811    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:04:41.012114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:41.012015    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:41.455679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:41.455646    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:04:41.457027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:41.028563    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.506641 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:41.372637    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.579542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:41.376624    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:04:41.735083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:41.028625    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.009162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.009054    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.085329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.064157    4282 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:04:42.085329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.064215    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.085329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.072851    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:04:42.085329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.072904    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"]
Oct 30 22:04:42.114528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.102939    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb" event={"ID":"ca26915d-a5a8-4111-aca6-54ee76f8cad9","Type":"ContainerStarted","Data":"804ba01c582d3bbfbd797b602d2646c9b651111bb2b8b875339bc640df5fec86"}
Oct 30 22:04:42.448806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.442696    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.523496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.522565    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.555642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.555594    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.600767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.600723    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:42.689922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.674204    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"]
Oct 30 22:04:42.689922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.674443    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" containerID="cri-o://4037dfde7c46dd9f1b416eb730c4c5a03c250c4b0100a98fc365bacb2f8f944f" gracePeriod=30
Oct 30 22:04:42.689922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.687587    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.775099 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.774877    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"]
Oct 30 22:04:42.777321 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.777291    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" containerID="cri-o://aaf9086352a7ea6b8412c49cf3b0b08aafb05fc4a42b6636005de0e40b558c0a" gracePeriod=30
Oct 30 22:04:42.782561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.782542    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:04:42.797742 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.783163    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:04:42.825520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:42.824316    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:04:43.543503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.499201    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:04:43.543503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.522839    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.570263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.570218    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.687881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.687833    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.738582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.738536    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:04:43.820017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.819918    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb" event={"ID":"ca26915d-a5a8-4111-aca6-54ee76f8cad9","Type":"ContainerStarted","Data":"848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b"}
Oct 30 22:04:43.820223 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.820020    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:04:43.820223 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.820102    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:04:43.820223 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.820128    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:43.820223 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.820187    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:04:43.820453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.820255    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:04:43.820453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.820312    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:04:43.846550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:43.846435    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb" podStartSLOduration=6.84641665 podStartE2EDuration="6.84641665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:43.837020501 +0000 UTC m=+1502.495116724" watchObservedRunningTime="2026-10-30 22:04:43.84641665 +0000 UTC m=+1502.504512879"
Oct 30 22:04:44.657604 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.657537    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:04:44.716242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.716212    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"]
Oct 30 22:04:44.721894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.721874    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.725068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.725044    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-cca1c7-serving-cert\""
Oct 30 22:04:44.732227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.732203    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"]
Oct 30 22:04:44.801500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.801454    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.801671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.801655    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7760b38b-dcfa-463f-969a-954a60353919-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.801727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.801718    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.801990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.801972    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrhhq\" (UniqueName: \"kubernetes.io/projected/7760b38b-dcfa-463f-969a-954a60353919-kube-api-access-wrhhq\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.915874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.915772    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.916061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.915958    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7760b38b-dcfa-463f-969a-954a60353919-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.916061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.916035    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.916200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.916149    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrhhq\" (UniqueName: \"kubernetes.io/projected/7760b38b-dcfa-463f-969a-954a60353919-kube-api-access-wrhhq\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.918486 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:04:44.916650    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-serving-cert: secret "maas-api-e2e-shared-a-w3-cca1c7-serving-cert" not found
Oct 30 22:04:44.918486 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:04:44.916716    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls podName:7760b38b-dcfa-463f-969a-954a60353919 nodeName:}" failed. No retries permitted until 2026-10-30 22:04:45.416697109 +0000 UTC m=+1504.074793315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls") pod "maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497" (UID: "7760b38b-dcfa-463f-969a-954a60353919") : secret "maas-api-e2e-shared-a-w3-cca1c7-serving-cert" not found
Oct 30 22:04:44.918486 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.917400    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7760b38b-dcfa-463f-969a-954a60353919-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.937224 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.932791    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:44.955240 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:44.955181    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrhhq\" (UniqueName: \"kubernetes.io/projected/7760b38b-dcfa-463f-969a-954a60353919-kube-api-access-wrhhq\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:45.428827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:45.428787    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:45.476095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:45.465393    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:45.645481 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:45.645447    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:45.935332 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:45.935300    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"]
Oct 30 22:04:45.936149 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:04:45.936123    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7760b38b_dcfa_463f_969a_954a60353919.slice/crio-801437547a534fb98d46478d11a7b7ae81d468b71042e9294d641b9a911a02de WatchSource:0}: Error finding container 801437547a534fb98d46478d11a7b7ae81d468b71042e9294d641b9a911a02de: Status 404 returned error can't find the container with id 801437547a534fb98d46478d11a7b7ae81d468b71042e9294d641b9a911a02de
Oct 30 22:04:46.852483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:46.852442    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497" event={"ID":"7760b38b-dcfa-463f-969a-954a60353919","Type":"ContainerStarted","Data":"09637470fd959ebd88e691b551ba4bfbb758f0fb29190ad722362dcb60b55bd5"}
Oct 30 22:04:46.852653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:46.852555    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497" event={"ID":"7760b38b-dcfa-463f-969a-954a60353919","Type":"ContainerStarted","Data":"801437547a534fb98d46478d11a7b7ae81d468b71042e9294d641b9a911a02de"}
Oct 30 22:04:46.852653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:46.852607    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:46.883991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:46.883946    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497" podStartSLOduration=2.488176276 podStartE2EDuration="2.883933624s" totalImagesPullingTime="395.757348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:46.878244675 +0000 UTC m=+1505.536340898" watchObservedRunningTime="2026-10-30 22:04:46.883933624 +0000 UTC m=+1505.542029875"
Oct 30 22:04:47.108624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:47.108549    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:04:47.861132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:47.861096    4282 generic.go:356] "Generic (PLEG): container finished" podID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerID="aaf9086352a7ea6b8412c49cf3b0b08aafb05fc4a42b6636005de0e40b558c0a" exitCode=0
Oct 30 22:04:47.861300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:47.861169    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" event={"ID":"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb","Type":"ContainerDied","Data":"aaf9086352a7ea6b8412c49cf3b0b08aafb05fc4a42b6636005de0e40b558c0a"}
Oct 30 22:04:48.013882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.013841    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-dh2j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:04:48.014196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.014145    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:04:48.018060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.018033    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-mwpd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 30 22:04:48.018207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.018073    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 30 22:04:48.516597 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.516563    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"]
Oct 30 22:04:48.522771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.522751    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.532206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.532181    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"]
Oct 30 22:04:48.567443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.567409    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.567580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.567487    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.567580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.567552    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2f586-028f-4202-8891-5fe93c3650d2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.567717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.567689    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sc7h\" (UniqueName: \"kubernetes.io/projected/caa2f586-028f-4202-8891-5fe93c3650d2-kube-api-access-2sc7h\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.588685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.588659    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ecfbd6-serving-cert\""
Oct 30 22:04:48.628633 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.620159    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 22:04:48.685726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.674023    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sc7h\" (UniqueName: \"kubernetes.io/projected/caa2f586-028f-4202-8891-5fe93c3650d2-kube-api-access-2sc7h\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.685726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.674327    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.685726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.674365    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.685726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.674425    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2f586-028f-4202-8891-5fe93c3650d2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.685726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.675327    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2f586-028f-4202-8891-5fe93c3650d2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.685726 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:04:48.675771    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-serving-cert: secret "maas-api-e2e-shared-a-w0-ecfbd6-serving-cert" not found
Oct 30 22:04:48.685726 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:04:48.675840    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls podName:caa2f586-028f-4202-8891-5fe93c3650d2 nodeName:}" failed. No retries permitted until 2026-10-30 22:04:49.175819399 +0000 UTC m=+1507.833915619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls") pod "maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7" (UID: "caa2f586-028f-4202-8891-5fe93c3650d2") : secret "maas-api-e2e-shared-a-w0-ecfbd6-serving-cert" not found
Oct 30 22:04:48.764547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.742812    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.769165    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sc7h\" (UniqueName: \"kubernetes.io/projected/caa2f586-028f-4202-8891-5fe93c3650d2-kube-api-access-2sc7h\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776014    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istiod-ca-cert\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776065    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-data\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776148    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-credential-socket\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776180    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-socket\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776274    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-podinfo\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776377    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-envoy\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776567    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-kube-api-access-mgz47\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-kube-api-access-mgz47\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776804    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-token\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-token\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.776883    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-certs\") pod \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\" (UID: \"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb\") "
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.778113    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-certs" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.778504    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istiod-ca-cert" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.778774    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-data" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.780960    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-credential-socket" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:48.786618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.781142    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-socket" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:48.790568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.790340    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-kube-api-access-mgz47" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "kube-api-access-mgz47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:48.799857 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.799833    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-token" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:48.806625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.806574    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-envoy" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:48.822759 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.822734    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-podinfo" pod "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" (UID: "72e5d66c-b1e7-4e49-99c7-1b0fb33598fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:04:48.879359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879269    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.879359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879309    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.879359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879324    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgz47\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-kube-api-access-mgz47\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.879359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879339    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.879359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879352    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.879359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879364    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.879913 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879376    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.879913 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879389    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.879913 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.879403    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:48.883202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.883176    4282 generic.go:356] "Generic (PLEG): container finished" podID="6b2fdf70-3c30-47a0-8f37-116848115739" containerID="4037dfde7c46dd9f1b416eb730c4c5a03c250c4b0100a98fc365bacb2f8f944f" exitCode=0
Oct 30 22:04:48.883320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.883244    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" event={"ID":"6b2fdf70-3c30-47a0-8f37-116848115739","Type":"ContainerDied","Data":"4037dfde7c46dd9f1b416eb730c4c5a03c250c4b0100a98fc365bacb2f8f944f"}
Oct 30 22:04:48.885023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.884994    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9" event={"ID":"72e5d66c-b1e7-4e49-99c7-1b0fb33598fb","Type":"ContainerDied","Data":"1d74ea6d9e7aaed0e4d5e248a11517d70a9cb4a1b1e459ff88b69dbf0fc84ef2"}
Oct 30 22:04:48.885137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.885061    4282 scope.go:122] "RemoveContainer" containerID="aaf9086352a7ea6b8412c49cf3b0b08aafb05fc4a42b6636005de0e40b558c0a"
Oct 30 22:04:48.885137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.885107    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"
Oct 30 22:04:48.914623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.914598    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"]
Oct 30 22:04:48.918555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.918531    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-mwpd9"]
Oct 30 22:04:48.949544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:48.949523    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 22:04:49.082110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082078    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-envoy\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.082274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082179    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b2fdf70-3c30-47a0-8f37-116848115739-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2fdf70-3c30-47a0-8f37-116848115739-istio-podinfo\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.082274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082201    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-credential-socket\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.082274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082242    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-certs\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.082388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082335    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-socket\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.082569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082546    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-kube-api-access-v7zkb\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-kube-api-access-v7zkb\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.082708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082558    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-credential-socket" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:49.082708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082606    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-certs" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:49.082708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082672    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-istio-token\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-istio-token\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.082875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082749    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-data\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.082875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082671    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-socket" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:49.082875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.082834    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2fdf70-3c30-47a0-8f37-116848115739-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2fdf70-3c30-47a0-8f37-116848115739-istiod-ca-cert\") pod \"6b2fdf70-3c30-47a0-8f37-116848115739\" (UID: \"6b2fdf70-3c30-47a0-8f37-116848115739\") "
Oct 30 22:04:49.083071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.083045    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-data" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:49.083687 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.083661    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2fdf70-3c30-47a0-8f37-116848115739-istiod-ca-cert" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:49.083955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.083898    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.083955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.083929    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.083955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.083943    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.084182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.083959    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.084182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.083973    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2fdf70-3c30-47a0-8f37-116848115739-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.084708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.084685    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b2fdf70-3c30-47a0-8f37-116848115739-istio-podinfo" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:04:49.084868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.084854    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-envoy" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:49.084963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.084944    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-kube-api-access-v7zkb" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "kube-api-access-v7zkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:49.085009 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.084988    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-istio-token" pod "6b2fdf70-3c30-47a0-8f37-116848115739" (UID: "6b2fdf70-3c30-47a0-8f37-116848115739"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:49.206871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.205881    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:49.206871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.206331    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2fdf70-3c30-47a0-8f37-116848115739-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.206871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.206353    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2fdf70-3c30-47a0-8f37-116848115739-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.206871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.206370    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7zkb\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-kube-api-access-v7zkb\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.206871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.206385    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2fdf70-3c30-47a0-8f37-116848115739-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:49.213523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.213441    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:49.445730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.445697    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:49.466542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.466514    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" path="/var/lib/kubelet/pods/72e5d66c-b1e7-4e49-99c7-1b0fb33598fb/volumes"
Oct 30 22:04:49.590732 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.584792    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:04:49.732663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.732640    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"]
Oct 30 22:04:49.733317 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:04:49.733295    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa2f586_028f_4202_8891_5fe93c3650d2.slice/crio-38b30aa43b8950669cf47b161290a16de21677955c1a73581c005cebe7daffaf WatchSource:0}: Error finding container 38b30aa43b8950669cf47b161290a16de21677955c1a73581c005cebe7daffaf: Status 404 returned error can't find the container with id 38b30aa43b8950669cf47b161290a16de21677955c1a73581c005cebe7daffaf
Oct 30 22:04:49.901173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.901137    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9" event={"ID":"6b2fdf70-3c30-47a0-8f37-116848115739","Type":"ContainerDied","Data":"2310563b8b16d30c30594cf151bd255d2be0ae76ffcb12e37ce1a01afb371400"}
Oct 30 22:04:49.901445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.901429    4282 scope.go:122] "RemoveContainer" containerID="4037dfde7c46dd9f1b416eb730c4c5a03c250c4b0100a98fc365bacb2f8f944f"
Oct 30 22:04:49.901684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.901670    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"
Oct 30 22:04:49.922595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.921787    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7" event={"ID":"caa2f586-028f-4202-8891-5fe93c3650d2","Type":"ContainerStarted","Data":"38b30aa43b8950669cf47b161290a16de21677955c1a73581c005cebe7daffaf"}
Oct 30 22:04:49.948044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.948016    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"]
Oct 30 22:04:49.954693 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:49.954666    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-dh2j9"]
Oct 30 22:04:50.930639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:50.930607    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7" event={"ID":"caa2f586-028f-4202-8891-5fe93c3650d2","Type":"ContainerStarted","Data":"72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2"}
Oct 30 22:04:50.930994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:50.930662    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:50.958221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:50.958169    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7" podStartSLOduration=2.397581422 podStartE2EDuration="2.958155595s" totalImagesPullingTime="560.574173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:50.954990505 +0000 UTC m=+1509.613086728" watchObservedRunningTime="2026-10-30 22:04:50.958155595 +0000 UTC m=+1509.616251817"
Oct 30 22:04:51.471865 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:51.471825    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" path="/var/lib/kubelet/pods/6b2fdf70-3c30-47a0-8f37-116848115739/volumes"
Oct 30 22:04:52.866593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:52.866563    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:04:54.788455 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.788424    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:04:54.950765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.950730    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"]
Oct 30 22:04:54.952248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.952226    4282 cpu_manager.go:414] "containerMap: removing container" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy"
Oct 30 22:04:54.952331 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.952251    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy"
Oct 30 22:04:54.952331 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.952321    4282 cpu_manager.go:414] "containerMap: removing container" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy"
Oct 30 22:04:54.952331 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.952329    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy"
Oct 30 22:04:54.952676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.952660    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e5d66c-b1e7-4e49-99c7-1b0fb33598fb" containerName="istio-proxy"
Oct 30 22:04:54.952723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.952696    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2fdf70-3c30-47a0-8f37-116848115739" containerName="istio-proxy"
Oct 30 22:04:54.959121 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.959079    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:54.966372 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.966344    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-38d6e2-openshift-default-dockercfg-gs5ql\""
Oct 30 22:04:54.967807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:54.967783    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"]
Oct 30 22:04:55.148659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.128525    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpcff\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-kube-api-access-vpcff\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.159581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.159546    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-envoy\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.159909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.159891    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-data\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.160112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.160097    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99d70de-0e2a-4574-b914-a667183840f6-istio-podinfo\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.166485 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.166438    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-socket\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.166720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.166701    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-credential-socket\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.166866 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.166851    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-istio-token\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.181593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.181548    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99d70de-0e2a-4574-b914-a667183840f6-istiod-ca-cert\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.181917 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.181876    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-certs\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.284728 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.284685    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-envoy\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.285616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.285592    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-data\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.302371 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.302341    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99d70de-0e2a-4574-b914-a667183840f6-istio-podinfo\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.303154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.286102    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-data\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.311570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.308834    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-envoy\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.312292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.312266    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99d70de-0e2a-4574-b914-a667183840f6-istio-podinfo\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.312545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.312523    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-socket\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.328530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.313068    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-credential-socket\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.334316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.313028    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-socket\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.334316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.318020    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-credential-socket\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.334316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.333694    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-istio-token\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.334316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.333750    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99d70de-0e2a-4574-b914-a667183840f6-istiod-ca-cert\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.334316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.333829    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-certs\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.334316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.334019    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpcff\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-kube-api-access-vpcff\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.344509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.335308    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99d70de-0e2a-4574-b914-a667183840f6-istiod-ca-cert\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.344509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.335585    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-certs\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.348192 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.348163    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpcff\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-kube-api-access-vpcff\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.348715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.348693    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-istio-token\") pod \"e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") " pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:55.580882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:55.580776    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:56.151085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:56.151060    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"]
Oct 30 22:04:56.152319 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:04:56.152296    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99d70de_0e2a_4574_b914_a667183840f6.slice/crio-9beae7ceaca96ad1581bf597fc2a03fdc400f10926d44afbbf6564eec9d456f9 WatchSource:0}: Error finding container 9beae7ceaca96ad1581bf597fc2a03fdc400f10926d44afbbf6564eec9d456f9: Status 404 returned error can't find the container with id 9beae7ceaca96ad1581bf597fc2a03fdc400f10926d44afbbf6564eec9d456f9
Oct 30 22:04:56.155175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:56.155137    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:56.155270 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:56.155215    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:56.155270 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:56.155248    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:04:56.942724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:56.942697    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:04:57.017814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:57.017780    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" event={"ID":"b99d70de-0e2a-4574-b914-a667183840f6","Type":"ContainerStarted","Data":"282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c"}
Oct 30 22:04:57.017982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:57.017832    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" event={"ID":"b99d70de-0e2a-4574-b914-a667183840f6","Type":"ContainerStarted","Data":"9beae7ceaca96ad1581bf597fc2a03fdc400f10926d44afbbf6564eec9d456f9"}
Oct 30 22:04:57.043131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:57.043065    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podStartSLOduration=3.043046176 podStartE2EDuration="3.043046176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:57.041730801 +0000 UTC m=+1515.699827025" watchObservedRunningTime="2026-10-30 22:04:57.043046176 +0000 UTC m=+1515.701142400"
Oct 30 22:04:57.581105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:57.581076    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:57.586874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:57.586853    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:58.025438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:58.025408    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:04:58.026540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:04:58.026517    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:05:00.104961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:00.104926    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:05:01.660320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:01.660280    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"]
Oct 30 22:05:01.980198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:01.965586    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:02.010887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.007800    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-38d6e2\""
Oct 30 22:05:02.078785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.019737    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-38d6e2-dockercfg-j9mg4\""
Oct 30 22:05:02.095954 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.084937    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbca5f55-1e74-457f-8f10-01e4601e8566-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c\" (UID: \"fbca5f55-1e74-457f-8f10-01e4601e8566\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:02.103533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.096430    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bs44\" (UniqueName: \"kubernetes.io/projected/fbca5f55-1e74-457f-8f10-01e4601e8566-kube-api-access-8bs44\") pod \"payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c\" (UID: \"fbca5f55-1e74-457f-8f10-01e4601e8566\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:02.324544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.321370    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bs44\" (UniqueName: \"kubernetes.io/projected/fbca5f55-1e74-457f-8f10-01e4601e8566-kube-api-access-8bs44\") pod \"payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c\" (UID: \"fbca5f55-1e74-457f-8f10-01e4601e8566\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:02.324544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.321637    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbca5f55-1e74-457f-8f10-01e4601e8566-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c\" (UID: \"fbca5f55-1e74-457f-8f10-01e4601e8566\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:02.324544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.322818    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbca5f55-1e74-457f-8f10-01e4601e8566-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c\" (UID: \"fbca5f55-1e74-457f-8f10-01e4601e8566\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:02.419825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.419784    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"]
Oct 30 22:05:02.585634 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.581873    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bs44\" (UniqueName: \"kubernetes.io/projected/fbca5f55-1e74-457f-8f10-01e4601e8566-kube-api-access-8bs44\") pod \"payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c\" (UID: \"fbca5f55-1e74-457f-8f10-01e4601e8566\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:02.616576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.611529    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:02.699518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.684689    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:02.795762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.795698    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:02.835369 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.835309    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 22:05:02.860584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:02.857614    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:03.571407 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.569591    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:03.592052 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.590801    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:05:03.759039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.758997    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:05:03.872897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.839821    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:05:03.872897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.796826    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:03.872897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.796764    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:03.872897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.845687    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:05:03.922642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.883209    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:03.957593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.930634    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 22:05:03.957593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.931912    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:03.957593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.942678    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:05:03.957593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:03.942740    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:05:04.349676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.349643    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"]
Oct 30 22:05:04.570492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.566813    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:04.570492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.569800    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:04.570492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.570133    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:05:04.581781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.579424    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-0a2577-openshift-default-dockercfg-6pss4\""
Oct 30 22:05:04.625568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.622004    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:05:04.625568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.622417    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:05:04.689495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.669973    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-token\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:04.870729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.869150    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-envoy\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:04.870729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.869315    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istiod-ca-cert\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:04.904576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.874122    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-podinfo\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:04.904576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.899701    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:04.904576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.841337    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:04.961921 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.789577    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:04.961921 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.758684    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:04.961921 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.840686    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:04.985529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.967817    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 30 22:05:04.986355 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.938567    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c47pf\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-kube-api-access-c47pf\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:04.987630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.987606    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.014499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.003402    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-data\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.080903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.080867    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-certs\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.081537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.081513    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-credential-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.082292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.987044    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"]
Oct 30 22:05:05.095189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.986584    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:05.095591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:04.995345    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": context deadline exceeded"
Oct 30 22:05:05.096079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.096060    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.231568    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-podinfo\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.231633    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47pf\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-kube-api-access-c47pf\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.231673    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.231887    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-data\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.231978    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-certs\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.232013    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-credential-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.232092    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-token\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.232165    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-envoy\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.238585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.232205    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istiod-ca-cert\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.240152 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.239683    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-data\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.267499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.246703    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-credential-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.267499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.247008    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-certs\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.267499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.247773    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-socket\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.329531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.272867    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istiod-ca-cert\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.329804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.315972    4282 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:05.361794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.361745    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:05.377661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.377628    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-envoy\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.378009 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.377988    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:05:05.446494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.426725    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-podinfo\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.598333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.591006    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:05:05.602298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.602263    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c47pf\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-kube-api-access-c47pf\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.758755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.629785    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:05.905199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.905109    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:05.951057 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:05:05.926287    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:05:05.951057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.947363    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"]
Oct 30 22:05:05.970942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.970909    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-token\") pod \"e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") " pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:05.971888 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:05.971853    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:06.000834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.000777    4282 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:06.001396 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.001371    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:06.062520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.032623    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:06.107036 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.104080    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:06.134318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.134288    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:05:06.261665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.254030    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:05:06.327057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.326962    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c" event={"ID":"fbca5f55-1e74-457f-8f10-01e4601e8566","Type":"ContainerStarted","Data":"7e7f205e5e09ecdd2f1da7abd74d415eaee21d43def91ab9d9789eaddbd4d171"}
Oct 30 22:05:06.636176 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.636064    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:06.637963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.637924    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:06.677644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.677595    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:05:06.952564 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.951643    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:06.980533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.979637    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:07.026652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:06.991711    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:05:07.117215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.117174    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:07.135171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.134698    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:05:07.138618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.138593    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:05:07.280974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.277518    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:05:07.487706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.487614    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c" event={"ID":"fbca5f55-1e74-457f-8f10-01e4601e8566","Type":"ContainerStarted","Data":"a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821"}
Oct 30 22:05:07.487706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.487679    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:07.511235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.511183    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c" podStartSLOduration=6.51114651 podStartE2EDuration="6.51114651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:07.507339224 +0000 UTC m=+1526.165435447" watchObservedRunningTime="2026-10-30 22:05:07.51114651 +0000 UTC m=+1526.169242736"
Oct 30 22:05:07.520340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.520312    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"]
Oct 30 22:05:07.524912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.524887    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:07.528378 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.528274    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0f73c9\""
Oct 30 22:05:07.529297 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.529275    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"]
Oct 30 22:05:07.530556 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.530535    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0f73c9-dockercfg-vl52z\""
Oct 30 22:05:07.561574 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.560943    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"]
Oct 30 22:05:07.583449 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:05:07.583416    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c6b36b_5c4f_4472_a233_9e6bbfdc1e6c.slice/crio-6d93d9c4482f816c4cea24e41b2480412c7e605fbc7fa882240a25fd09fb0a4c WatchSource:0}: Error finding container 6d93d9c4482f816c4cea24e41b2480412c7e605fbc7fa882240a25fd09fb0a4c: Status 404 returned error can't find the container with id 6d93d9c4482f816c4cea24e41b2480412c7e605fbc7fa882240a25fd09fb0a4c
Oct 30 22:05:07.587268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.586709    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:07.587268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.586797    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:07.587268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.586828    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:07.687108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.687079    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj7bz\" (UniqueName: \"kubernetes.io/projected/3900d25b-7d5e-4b35-826c-afaec7394e5b-kube-api-access-mj7bz\") pod \"payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p\" (UID: \"3900d25b-7d5e-4b35-826c-afaec7394e5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:07.687758 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.687721    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3900d25b-7d5e-4b35-826c-afaec7394e5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p\" (UID: \"3900d25b-7d5e-4b35-826c-afaec7394e5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:08.005551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:07.790598    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj7bz\" (UniqueName: \"kubernetes.io/projected/3900d25b-7d5e-4b35-826c-afaec7394e5b-kube-api-access-mj7bz\") pod \"payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p\" (UID: \"3900d25b-7d5e-4b35-826c-afaec7394e5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:08.023491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:08.006299    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3900d25b-7d5e-4b35-826c-afaec7394e5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p\" (UID: \"3900d25b-7d5e-4b35-826c-afaec7394e5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:08.024088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:08.019354    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3900d25b-7d5e-4b35-826c-afaec7394e5b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p\" (UID: \"3900d25b-7d5e-4b35-826c-afaec7394e5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:08.089632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:08.089130    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj7bz\" (UniqueName: \"kubernetes.io/projected/3900d25b-7d5e-4b35-826c-afaec7394e5b-kube-api-access-mj7bz\") pod \"payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p\" (UID: \"3900d25b-7d5e-4b35-826c-afaec7394e5b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:08.144271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:08.144239    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:08.499757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:08.499719    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" event={"ID":"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c","Type":"ContainerStarted","Data":"7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da"}
Oct 30 22:05:08.499967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:08.499785    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" event={"ID":"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c","Type":"ContainerStarted","Data":"6d93d9c4482f816c4cea24e41b2480412c7e605fbc7fa882240a25fd09fb0a4c"}
Oct 30 22:05:08.522935 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:08.522841    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" podStartSLOduration=5.522826215 podStartE2EDuration="5.522826215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:08.520880613 +0000 UTC m=+1527.178976835" watchObservedRunningTime="2026-10-30 22:05:08.522826215 +0000 UTC m=+1527.180922438"
Oct 30 22:05:08.686935 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:08.686902    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"]
Oct 30 22:05:08.687713 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:05:08.687687    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3900d25b_7d5e_4b35_826c_afaec7394e5b.slice/crio-4ea64eb827fa91f68d1f6fb28cdc6769a187f70453ea3723c97cefb056d12f3c WatchSource:0}: Error finding container 4ea64eb827fa91f68d1f6fb28cdc6769a187f70453ea3723c97cefb056d12f3c: Status 404 returned error can't find the container with id 4ea64eb827fa91f68d1f6fb28cdc6769a187f70453ea3723c97cefb056d12f3c
Oct 30 22:05:09.001430 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:09.001400    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:09.007723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:09.007698    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:09.507985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:09.507945    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p" event={"ID":"3900d25b-7d5e-4b35-826c-afaec7394e5b","Type":"ContainerStarted","Data":"333bae14caaf449017644aa3b156abb342fb20d581eed251bde12d44445939be"}
Oct 30 22:05:09.508164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:09.508143    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:09.508267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:09.508189    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p" event={"ID":"3900d25b-7d5e-4b35-826c-afaec7394e5b","Type":"ContainerStarted","Data":"4ea64eb827fa91f68d1f6fb28cdc6769a187f70453ea3723c97cefb056d12f3c"}
Oct 30 22:05:09.508443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:09.508423    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:09.510049 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:09.510026    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:05:09.527586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:09.527540    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p" podStartSLOduration=2.5275282 podStartE2EDuration="2.5275282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:09.525191782 +0000 UTC m=+1528.183288005" watchObservedRunningTime="2026-10-30 22:05:09.5275282 +0000 UTC m=+1528.185624422"
Oct 30 22:05:12.326771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.326744    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"]
Oct 30 22:05:12.332415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.332394    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:12.346583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.346552    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-0a2577\""
Oct 30 22:05:12.351275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.351249    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"]
Oct 30 22:05:12.362358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.362330    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-0a2577-dockercfg-jsbdj\""
Oct 30 22:05:12.522514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.520174    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpvs\" (UniqueName: \"kubernetes.io/projected/cc889df4-c49f-4b23-9db5-2561028d5b2e-kube-api-access-jwpvs\") pod \"payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn\" (UID: \"cc889df4-c49f-4b23-9db5-2561028d5b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:12.522514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.520287    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc889df4-c49f-4b23-9db5-2561028d5b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn\" (UID: \"cc889df4-c49f-4b23-9db5-2561028d5b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:12.650545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.631519    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpvs\" (UniqueName: \"kubernetes.io/projected/cc889df4-c49f-4b23-9db5-2561028d5b2e-kube-api-access-jwpvs\") pod \"payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn\" (UID: \"cc889df4-c49f-4b23-9db5-2561028d5b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:12.650545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.631692    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc889df4-c49f-4b23-9db5-2561028d5b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn\" (UID: \"cc889df4-c49f-4b23-9db5-2561028d5b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:12.650545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.638798    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc889df4-c49f-4b23-9db5-2561028d5b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn\" (UID: \"cc889df4-c49f-4b23-9db5-2561028d5b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:12.670493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.662144    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpvs\" (UniqueName: \"kubernetes.io/projected/cc889df4-c49f-4b23-9db5-2561028d5b2e-kube-api-access-jwpvs\") pod \"payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn\" (UID: \"cc889df4-c49f-4b23-9db5-2561028d5b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:12.953675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:12.953640    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:13.375292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:13.336926    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"]
Oct 30 22:05:13.623250 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:13.622873    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn" event={"ID":"cc889df4-c49f-4b23-9db5-2561028d5b2e","Type":"ContainerStarted","Data":"6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836"}
Oct 30 22:05:13.623250 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:13.622945    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn" event={"ID":"cc889df4-c49f-4b23-9db5-2561028d5b2e","Type":"ContainerStarted","Data":"85909a548cd0efcf8ee5feb45791383a7ae5e2fc39ed486253be91574e26da07"}
Oct 30 22:05:13.623250 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:13.622975    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:13.648430 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:13.648356    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn" podStartSLOduration=1.6483369639999998 podStartE2EDuration="1.648336964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:13.647542022 +0000 UTC m=+1532.305638246" watchObservedRunningTime="2026-10-30 22:05:13.648336964 +0000 UTC m=+1532.306433188"
Oct 30 22:05:14.917065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.917027    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"]
Oct 30 22:05:14.923907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.923879    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942091    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942153    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942188    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnkj2\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-kube-api-access-gnkj2\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942306    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942340    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942383    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942571    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942629    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.944513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.942656    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:14.945780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.945749    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cca1c7-openshift-default-dockercfg-lmtw8\""
Oct 30 22:05:14.947189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:14.947163    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"]
Oct 30 22:05:15.044190 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044154    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.044359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044224    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.044359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044258    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.044359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044325    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.044605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044368    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.044605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044395    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnkj2\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-kube-api-access-gnkj2\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.044605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044538    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.044605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044576    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.044840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.044658    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.045057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.045031    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.045350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.045323    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.046117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.045612    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.046117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.045680    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.046117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.045724    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.049126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.049099    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.049885 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.049868    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.061940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.061918    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.062063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.061983    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnkj2\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-kube-api-access-gnkj2\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.276557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.268009    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:15.972653 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:05:15.972625    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a92ac34_b27e_4d6d_8ed2_27f100247b6f.slice/crio-5184fe54ca4f8073a55d5327f67d899c57e70f358fe9bbcea6a05f54eab75142 WatchSource:0}: Error finding container 5184fe54ca4f8073a55d5327f67d899c57e70f358fe9bbcea6a05f54eab75142: Status 404 returned error can't find the container with id 5184fe54ca4f8073a55d5327f67d899c57e70f358fe9bbcea6a05f54eab75142
Oct 30 22:05:15.973513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.973482    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"]
Oct 30 22:05:15.976439 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.976395    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:15.976561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.976541    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:15.976618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:15.976599    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:16.816579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.802375    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" event={"ID":"3a92ac34-b27e-4d6d-8ed2-27f100247b6f","Type":"ContainerStarted","Data":"9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530"}
Oct 30 22:05:16.816579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.802451    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" event={"ID":"3a92ac34-b27e-4d6d-8ed2-27f100247b6f","Type":"ContainerStarted","Data":"5184fe54ca4f8073a55d5327f67d899c57e70f358fe9bbcea6a05f54eab75142"}
Oct 30 22:05:16.819061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.817779    4282 generic.go:356] "Generic (PLEG): container finished" podID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerID="df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d" exitCode=0
Oct 30 22:05:16.819061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.817828    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" event={"ID":"f3b514a9-b9c4-4705-a789-a7acaba6a27f","Type":"ContainerDied","Data":"df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d"}
Oct 30 22:05:16.819061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.818397    4282 scope.go:122] "RemoveContainer" containerID="df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d"
Oct 30 22:05:16.831214 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.830026    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:16.831214 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.830141    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:16.831214 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.830199    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:16.889695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:16.889594    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" podStartSLOduration=2.889574271 podStartE2EDuration="2.889574271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:16.883869508 +0000 UTC m=+1535.541965744" watchObservedRunningTime="2026-10-30 22:05:16.889574271 +0000 UTC m=+1535.547670495"
Oct 30 22:05:17.289864 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:17.289827    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:17.291433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:17.291399    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:17.291616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:17.291476    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 22:05:17.850189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:17.850147    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" event={"ID":"f3b514a9-b9c4-4705-a789-a7acaba6a27f","Type":"ContainerStarted","Data":"7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e"}
Oct 30 22:05:18.015435 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.015394    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"]
Oct 30 22:05:18.020046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.020025    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.031927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.031901    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"]
Oct 30 22:05:18.043965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.043937    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-5313f0-openshift-default-dockercfg-scpj8\""
Oct 30 22:05:18.116247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116132    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.116247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116227    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q6q8\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-kube-api-access-2q6q8\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.116518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116274    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.116518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116319    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.116518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116351    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.116518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116385    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.116518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116422    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.116801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116741    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.116801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.116777    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.217961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.217925    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.217961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.217962    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.218163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218066    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.218163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218123    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q6q8\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-kube-api-access-2q6q8\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.218235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218170    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.218448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218420    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.218523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218204    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.218588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218553    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.218818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218760    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.218818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218774    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.219064 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218952    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.219064 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.218993    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.219267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.219245    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.219338 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.219269    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.220602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.220579    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.224612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.224592    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.236065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.236043    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q6q8\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-kube-api-access-2q6q8\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.246005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.245982    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.305627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.305590    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:18.306135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.305641    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 22:05:18.339107 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.339076    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:18.511656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.510296    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:05:18.626377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.626295    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:05:18.627275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.626980    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:05:18.627275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.627018    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 22:05:18.734352 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:05:18.734321    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8ca6ab_097b_4e0b_a2f1_584306642c62.slice/crio-bf02ce0d3c29e06bd17d122cf021748cc556aea0d4326082fb792afa7f6d53ad WatchSource:0}: Error finding container bf02ce0d3c29e06bd17d122cf021748cc556aea0d4326082fb792afa7f6d53ad: Status 404 returned error can't find the container with id bf02ce0d3c29e06bd17d122cf021748cc556aea0d4326082fb792afa7f6d53ad
Oct 30 22:05:18.736260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.736232    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"]
Oct 30 22:05:18.737484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.737438    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:18.737602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.737538    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:18.737602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.737572    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:18.899501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:18.894105    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" event={"ID":"ac8ca6ab-097b-4e0b-a2f1-584306642c62","Type":"ContainerStarted","Data":"bf02ce0d3c29e06bd17d122cf021748cc556aea0d4326082fb792afa7f6d53ad"}
Oct 30 22:05:19.329550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:19.324071    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:19.329550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:19.324127    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 22:05:19.627535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:19.626498    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:05:19.627535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:19.626559    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 22:05:19.933406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:19.932938    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" event={"ID":"ac8ca6ab-097b-4e0b-a2f1-584306642c62","Type":"ContainerStarted","Data":"90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c"}
Oct 30 22:05:19.984040 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:19.983979    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podStartSLOduration=2.9839578490000003 podStartE2EDuration="2.983957849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:19.95800185 +0000 UTC m=+1538.616098073" watchObservedRunningTime="2026-10-30 22:05:19.983957849 +0000 UTC m=+1538.642054072"
Oct 30 22:05:20.294564 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.292854    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:20.294564 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.292920    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 22:05:20.339486 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.339443    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:20.340538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.340514    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:05:20.340627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.340563    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 22:05:20.564350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.563682    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:05:20.641493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.628718    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:05:20.641493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.628778    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 22:05:20.782868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:20.782825    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:05:21.297492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:21.296977    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:21.297492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:21.297040    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 22:05:21.351509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:21.343988    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:05:21.351509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:21.344066    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 22:05:21.628698 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:21.628608    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:05:21.628966 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:21.628939    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 22:05:22.292207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:22.291822    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 22:05:22.292207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:22.291885    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 22:05:22.342942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:22.342833    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:05:22.342942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:22.342887    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 22:05:22.630566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:22.628829    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:05:22.630566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:22.628885    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 22:05:23.324067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:23.324037    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:23.324493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:23.324446    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:23.325943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:23.325906    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:05:23.340369 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:23.340344    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:05:23.340592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:23.340398    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 22:05:23.626761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:23.626680    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 22:05:23.626761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:23.626732    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 22:05:24.340280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:24.340246    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:05:24.340709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:24.340303    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 22:05:24.641329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:24.639814    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:05:24.646144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:24.646118    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:05:24.647430 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:24.647407    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:05:25.348529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:25.348425    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 22:05:25.349243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:25.349207    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 22:05:26.409859 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.409827    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:26.411248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.411222    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:26.412522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.412500    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:05:26.751301 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.751252    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"]
Oct 30 22:05:26.757883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.757861    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:26.781374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.781351    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"]
Oct 30 22:05:26.787236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.787209    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-9c8ecf-serving-cert\""
Oct 30 22:05:26.914561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.914528    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wng4\" (UniqueName: \"kubernetes.io/projected/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-kube-api-access-7wng4\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:26.914810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.914792    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:26.914905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.914892    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:26.914998 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:26.914976    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.020737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.020640    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.021326 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.021291    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.068772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.068735    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.069843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.069819    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wng4\" (UniqueName: \"kubernetes.io/projected/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-kube-api-access-7wng4\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.070509 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:05:27.021059    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-serving-cert: secret "maas-api-e2e-shared-b-w2-9c8ecf-serving-cert" not found
Oct 30 22:05:27.070634 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:05:27.070544    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls podName:5dc97cfc-917f-4a48-8651-ef29ebaf4b56 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:27.570521017 +0000 UTC m=+1546.228617221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls") pod "maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf" (UID: "5dc97cfc-917f-4a48-8651-ef29ebaf4b56") : secret "maas-api-e2e-shared-b-w2-9c8ecf-serving-cert" not found
Oct 30 22:05:27.070809 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.055133    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.071006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.069752    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.109621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.109589    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wng4\" (UniqueName: \"kubernetes.io/projected/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-kube-api-access-7wng4\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.589635 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.589584    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.594291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.594226    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:27.675818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:27.675780    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:28.009337 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:28.009307    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"]
Oct 30 22:05:28.009461 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:05:28.009377    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc97cfc_917f_4a48_8651_ef29ebaf4b56.slice/crio-27e2aa0a3c4a9f0feab6662f7945381d48f4af75d3763f2b1a78d58f1e7611dc WatchSource:0}: Error finding container 27e2aa0a3c4a9f0feab6662f7945381d48f4af75d3763f2b1a78d58f1e7611dc: Status 404 returned error can't find the container with id 27e2aa0a3c4a9f0feab6662f7945381d48f4af75d3763f2b1a78d58f1e7611dc
Oct 30 22:05:28.436500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:28.430102    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf" event={"ID":"5dc97cfc-917f-4a48-8651-ef29ebaf4b56","Type":"ContainerStarted","Data":"27e2aa0a3c4a9f0feab6662f7945381d48f4af75d3763f2b1a78d58f1e7611dc"}
Oct 30 22:05:29.438288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:29.438249    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf" event={"ID":"5dc97cfc-917f-4a48-8651-ef29ebaf4b56","Type":"ContainerStarted","Data":"b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d"}
Oct 30 22:05:29.438710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:29.438317    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:29.460449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:29.460396    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf" podStartSLOduration=2.934570089 podStartE2EDuration="3.460379517s" totalImagesPullingTime="525.809428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:29.458827824 +0000 UTC m=+1548.116924047" watchObservedRunningTime="2026-10-30 22:05:29.460379517 +0000 UTC m=+1548.118475741"
Oct 30 22:05:34.303565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:34.303530    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:05:34.700390 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:05:34.700356    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:05:35.469632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:35.469590    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:05:42.666431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:42.666365    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"]
Oct 30 22:05:42.667018 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:42.666802    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy" containerID="cri-o://2b27b229f1a74197af93fbfebe56eb97366ae559ac88a1ab259af280eeddb65e" gracePeriod=30
Oct 30 22:05:42.987527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:42.987491    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"]
Oct 30 22:05:42.994560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:42.994528    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.007247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.003882    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cca1c7\"/\"openshift-service-ca.crt\""
Oct 30 22:05:43.007247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.003975    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"]
Oct 30 22:05:43.007247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.004131    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cca1c7\"/\"kube-root-ca.crt\""
Oct 30 22:05:43.007247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.004248    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cca1c7\"/\"default-dockercfg-ggc2c\""
Oct 30 22:05:43.007247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.004365    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cca1c7\"/\"auth-test-model-cca1c7-kserve-self-signed-certs\""
Oct 30 22:05:43.067565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.067520    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-tmp-dir\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.067769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.067700    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-dshm\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.067844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.067798    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-model-cache\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.067951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.067927    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-home\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.068030 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.068016    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80767346-e0df-456c-8acf-fe93780e238f-tls-certs\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.068091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.068043    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtspn\" (UniqueName: \"kubernetes.io/projected/80767346-e0df-456c-8acf-fe93780e238f-kube-api-access-vtspn\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.178840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.175748    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-model-cache\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.178840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.175918    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-home\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.178840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.175998    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80767346-e0df-456c-8acf-fe93780e238f-tls-certs\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.178840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.176025    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtspn\" (UniqueName: \"kubernetes.io/projected/80767346-e0df-456c-8acf-fe93780e238f-kube-api-access-vtspn\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.178840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.176203    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-tmp-dir\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.178840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.176367    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-dshm\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.225123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.199402    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-model-cache\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.225123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.200005    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-home\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.225123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.200269    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-tmp-dir\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.225123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.208823    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-dshm\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.225123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.216186    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80767346-e0df-456c-8acf-fe93780e238f-tls-certs\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.272561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.269286    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtspn\" (UniqueName: \"kubernetes.io/projected/80767346-e0df-456c-8acf-fe93780e238f-kube-api-access-vtspn\") pod \"auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") " pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.322410 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.322070    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:43.710420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.710391    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"]
Oct 30 22:05:43.713025 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:05:43.712993    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80767346_e0df_456c_8acf_fe93780e238f.slice/crio-4c7bd2d9002ba55b21e31aeafa83f4242986b74b557dfe175c3d2fdcc028e5c8 WatchSource:0}: Error finding container 4c7bd2d9002ba55b21e31aeafa83f4242986b74b557dfe175c3d2fdcc028e5c8: Status 404 returned error can't find the container with id 4c7bd2d9002ba55b21e31aeafa83f4242986b74b557dfe175c3d2fdcc028e5c8
Oct 30 22:05:43.857393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.857359    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm" event={"ID":"80767346-e0df-456c-8acf-fe93780e238f","Type":"ContainerStarted","Data":"a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f"}
Oct 30 22:05:43.857604 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.857405    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm" event={"ID":"80767346-e0df-456c-8acf-fe93780e238f","Type":"ContainerStarted","Data":"4c7bd2d9002ba55b21e31aeafa83f4242986b74b557dfe175c3d2fdcc028e5c8"}
Oct 30 22:05:43.876522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:43.876422    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm" podStartSLOduration=1.876406311 podStartE2EDuration="1.876406311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:43.874024112 +0000 UTC m=+1562.532120336" watchObservedRunningTime="2026-10-30 22:05:43.876406311 +0000 UTC m=+1562.534502538"
Oct 30 22:05:44.067793 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:44.067656    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:05:44.067793 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:44.067755    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:05:44.323432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:44.323371    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:45.592817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:45.592787    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:05:45.593324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:45.592858    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:05:46.884992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:46.884962    4282 generic.go:356] "Generic (PLEG): container finished" podID="b99d70de-0e2a-4574-b914-a667183840f6" containerID="282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c" exitCode=0
Oct 30 22:05:46.885442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:46.885035    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" event={"ID":"b99d70de-0e2a-4574-b914-a667183840f6","Type":"ContainerDied","Data":"282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c"}
Oct 30 22:05:46.885442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:46.885435    4282 scope.go:122] "RemoveContainer" containerID="282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c"
Oct 30 22:05:46.887079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:46.887047    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:46.887177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:46.887120    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:46.887177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:46.887158    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:05:47.301273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.298859    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"]
Oct 30 22:05:47.304573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.304549    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"]
Oct 30 22:05:47.304794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.304632    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.308135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.308111    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ecfbd6\"/\"auth-test-model-ecfbd6-kserve-self-signed-certs\""
Oct 30 22:05:47.308407 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.308389    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-ecfbd6\"/\"default-dockercfg-q9cf9\""
Oct 30 22:05:47.308886 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.308861    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ecfbd6\"/\"kube-root-ca.crt\""
Oct 30 22:05:47.308994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.308937    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-ecfbd6\"/\"openshift-service-ca.crt\""
Oct 30 22:05:47.398828 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.398797    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c266b897-204e-44bb-94ba-1e20c7013b5c-tls-certs\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.398992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.398849    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-dshm\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.399038 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.399015    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-home\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.399171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.399151    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mbf\" (UniqueName: \"kubernetes.io/projected/c266b897-204e-44bb-94ba-1e20c7013b5c-kube-api-access-s4mbf\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.399307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.399294    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-tmp-dir\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.399351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.399318    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-model-cache\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.524628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.512166    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c266b897-204e-44bb-94ba-1e20c7013b5c-tls-certs\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.530130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.530100    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-dshm\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.566924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.566841    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-home\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.572781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.567295    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-home\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.572781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.566865    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-dshm\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.572781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.567325    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mbf\" (UniqueName: \"kubernetes.io/projected/c266b897-204e-44bb-94ba-1e20c7013b5c-kube-api-access-s4mbf\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.572781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.567597    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-tmp-dir\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.572781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.567631    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-model-cache\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.588072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.587744    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-tmp-dir\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.588072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.587992    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-model-cache\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.617491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.604818    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c266b897-204e-44bb-94ba-1e20c7013b5c-tls-certs\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.617491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.605754    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mbf\" (UniqueName: \"kubernetes.io/projected/c266b897-204e-44bb-94ba-1e20c7013b5c-kube-api-access-s4mbf\") pod \"auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") " pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.627525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.623771    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:47.708877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.701923    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"]
Oct 30 22:05:47.735510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.735438    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"]
Oct 30 22:05:47.915515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.915381    4282 generic.go:356] "Generic (PLEG): container finished" podID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerID="2b27b229f1a74197af93fbfebe56eb97366ae559ac88a1ab259af280eeddb65e" exitCode=0
Oct 30 22:05:47.915515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.915456    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" event={"ID":"2a8d8505-4b73-4308-b7eb-4a778777e5d1","Type":"ContainerDied","Data":"2b27b229f1a74197af93fbfebe56eb97366ae559ac88a1ab259af280eeddb65e"}
Oct 30 22:05:47.915515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.915518    4282 scope.go:122] "RemoveContainer" containerID="a6b8046ebdfbce6552775e7ed952654051fe5a5fdcb11f847e44edeca29e1ad0"
Oct 30 22:05:47.922096 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:47.922058    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" event={"ID":"b99d70de-0e2a-4574-b914-a667183840f6","Type":"ContainerStarted","Data":"95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a"}
Oct 30 22:05:48.019326 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:05:48.019291    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc266b897_204e_44bb_94ba_1e20c7013b5c.slice/crio-274e03de42f3a9f801c7365513c3a8640f3b4de414d0bf0f63b41b0d9ac7fbfd WatchSource:0}: Error finding container 274e03de42f3a9f801c7365513c3a8640f3b4de414d0bf0f63b41b0d9ac7fbfd: Status 404 returned error can't find the container with id 274e03de42f3a9f801c7365513c3a8640f3b4de414d0bf0f63b41b0d9ac7fbfd
Oct 30 22:05:48.045605 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.028936    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"]
Oct 30 22:05:48.509674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.509609    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 22:05:48.583919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.583882    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:05:48.585004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.584972    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:05:48.585135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.585027    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:05:48.683489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683409    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-kube-api-access-8wl59\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-kube-api-access-8wl59\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.683707 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683531    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-data\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.683707 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683602    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-token\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-token\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.683707 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683637    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-podinfo\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.683707 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683682    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istiod-ca-cert\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.683955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683721    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-certs\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.683955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683762    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-socket\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.683955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683790    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-credential-socket\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.683955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.683852    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-envoy\") pod \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\" (UID: \"2a8d8505-4b73-4308-b7eb-4a778777e5d1\") "
Oct 30 22:05:48.687739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.687708    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-data" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:48.687926 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.687907    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-certs" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:48.689304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.689252    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-socket" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:48.689775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.689723    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.689775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.689768    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.690004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.689783    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.690072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.690053    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-credential-socket" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:48.690183 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.690155    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istiod-ca-cert" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:48.690907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.690885    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-podinfo" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:05:48.692458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.692436    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-envoy" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:48.695396 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.695355    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-token" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:48.696991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.696968    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-kube-api-access-8wl59" pod "2a8d8505-4b73-4308-b7eb-4a778777e5d1" (UID: "2a8d8505-4b73-4308-b7eb-4a778777e5d1"). InnerVolumeSpecName "kube-api-access-8wl59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:48.791222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.791136    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.791222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.791175    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.791222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.791187    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wl59\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-kube-api-access-8wl59\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.791222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.791197    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.791222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.791208    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.791222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.791217    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8d8505-4b73-4308-b7eb-4a778777e5d1-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:48.986952 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.986901    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4" event={"ID":"c266b897-204e-44bb-94ba-1e20c7013b5c","Type":"ContainerStarted","Data":"6054a8069dcc44b269a0aa71845c88d80b824ccc8e6dbd140d41e240b6c36567"}
Oct 30 22:05:48.987506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.986964    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4" event={"ID":"c266b897-204e-44bb-94ba-1e20c7013b5c","Type":"ContainerStarted","Data":"274e03de42f3a9f801c7365513c3a8640f3b4de414d0bf0f63b41b0d9ac7fbfd"}
Oct 30 22:05:48.989852 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.989831    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"
Oct 30 22:05:48.989942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.989865    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t" event={"ID":"2a8d8505-4b73-4308-b7eb-4a778777e5d1","Type":"ContainerDied","Data":"e43b356e3af5476b230c4274da2b7527b408843347611206b92510d0186439c4"}
Oct 30 22:05:48.989942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:48.989911    4282 scope.go:122] "RemoveContainer" containerID="2b27b229f1a74197af93fbfebe56eb97366ae559ac88a1ab259af280eeddb65e"
Oct 30 22:05:49.008142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:49.008095    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4" podStartSLOduration=2.008051692 podStartE2EDuration="2.008051692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:49.005199018 +0000 UTC m=+1567.663295239" watchObservedRunningTime="2026-10-30 22:05:49.008051692 +0000 UTC m=+1567.666147971"
Oct 30 22:05:49.021454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:49.021430    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"]
Oct 30 22:05:49.027880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:49.027862    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-wbd8t"]
Oct 30 22:05:49.485521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:49.485483    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" path="/var/lib/kubelet/pods/2a8d8505-4b73-4308-b7eb-4a778777e5d1/volumes"
Oct 30 22:05:49.582300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:49.582268    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:05:49.582514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:49.582319    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:05:49.627405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:49.626579    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:50.581564 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:50.581535    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:05:50.581985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:50.581595    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:05:51.582013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:51.581976    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:05:51.583022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:51.582035    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:05:52.581923 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:52.581883    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:05:52.582136 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:52.581955    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:05:52.706949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:52.706876    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq" podUID="38b8e600-068c-413b-b73c-244dd27e942c" containerName="payload-processing" containerID="cri-o://eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28" gracePeriod=25
Oct 30 22:05:52.922164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:52.922105    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz" podUID="5dc521f2-3dfa-470a-b780-ee43a2a113c4" containerName="payload-pre-processing" containerID="cri-o://d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60" gracePeriod=25
Oct 30 22:05:53.339514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:53.337157    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:53.383080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:53.383050    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:53.581445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:53.581408    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 30 22:05:53.581625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:53.581492    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 30 22:05:54.042833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.036021    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"]
Oct 30 22:05:54.042833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.037167    4282 cpu_manager.go:414] "containerMap: removing container" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy"
Oct 30 22:05:54.042833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.037190    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy"
Oct 30 22:05:54.042833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.037222    4282 cpu_manager.go:414] "containerMap: removing container" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy"
Oct 30 22:05:54.042833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.037230    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy"
Oct 30 22:05:54.042833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.037574    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy"
Oct 30 22:05:54.042833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.037594    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8d8505-4b73-4308-b7eb-4a778777e5d1" containerName="istio-proxy"
Oct 30 22:05:54.045940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.045917    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"]
Oct 30 22:05:54.046088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.046024    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.049291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.049162    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-5313f0\"/\"default-dockercfg-bd2vz\""
Oct 30 22:05:54.049291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.049180    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-5313f0\"/\"kube-root-ca.crt\""
Oct 30 22:05:54.049291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.049180    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-5313f0\"/\"auth-test-model-5313f0-kserve-self-signed-certs\""
Oct 30 22:05:54.049291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.049268    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-5313f0\"/\"openshift-service-ca.crt\""
Oct 30 22:05:54.075142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.075123    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:05:54.098640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.098612    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-tmp-dir\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.098754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.098685    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdnnh\" (UniqueName: \"kubernetes.io/projected/66562807-8fd2-448d-8187-95729caa75f7-kube-api-access-wdnnh\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.098918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.098895    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-dshm\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.099057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.099039    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66562807-8fd2-448d-8187-95729caa75f7-tls-certs\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.099102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.099087    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-model-cache\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.099165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.099150    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-home\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.200437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.200408    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-dshm\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.200677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.200555    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66562807-8fd2-448d-8187-95729caa75f7-tls-certs\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.200677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.200597    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-model-cache\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.200677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.200622    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-home\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.200851 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.200695    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-tmp-dir\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.201107 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.201055    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-tmp-dir\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.201107 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.201095    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-home\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.201265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.201158    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-model-cache\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.201265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.200718    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdnnh\" (UniqueName: \"kubernetes.io/projected/66562807-8fd2-448d-8187-95729caa75f7-kube-api-access-wdnnh\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.203157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.203135    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-dshm\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.203380 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.203362    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66562807-8fd2-448d-8187-95729caa75f7-tls-certs\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.236310 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.236243    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdnnh\" (UniqueName: \"kubernetes.io/projected/66562807-8fd2-448d-8187-95729caa75f7-kube-api-access-wdnnh\") pod \"auth-test-model-5313f0-kserve-54c5885fcc-hnkj4\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") " pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.371508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.371343    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:54.586633 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.586603    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:05:54.587067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.587048    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:05:54.588006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.587987    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:05:54.771261 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:05:54.771213    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66562807_8fd2_448d_8187_95729caa75f7.slice/crio-4babbb721a68a705050e2922028b0a1708f3f326287046baa65d53ad99e66aa1 WatchSource:0}: Error finding container 4babbb721a68a705050e2922028b0a1708f3f326287046baa65d53ad99e66aa1: Status 404 returned error can't find the container with id 4babbb721a68a705050e2922028b0a1708f3f326287046baa65d53ad99e66aa1
Oct 30 22:05:54.773224 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:54.773138    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"]
Oct 30 22:05:55.069576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:55.069491    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4" event={"ID":"66562807-8fd2-448d-8187-95729caa75f7","Type":"ContainerStarted","Data":"3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605"}
Oct 30 22:05:55.069576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:55.069540    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4" event={"ID":"66562807-8fd2-448d-8187-95729caa75f7","Type":"ContainerStarted","Data":"4babbb721a68a705050e2922028b0a1708f3f326287046baa65d53ad99e66aa1"}
Oct 30 22:05:55.090747 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:55.090701    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4" podStartSLOduration=2.090688409 podStartE2EDuration="2.090688409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:55.087768858 +0000 UTC m=+1573.745865080" watchObservedRunningTime="2026-10-30 22:05:55.090688409 +0000 UTC m=+1573.748784631"
Oct 30 22:05:55.376672 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:55.376588    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:05:56.188778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:56.183029    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz" podUID="5dc521f2-3dfa-470a-b780-ee43a2a113c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:05:56.212499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:56.210577    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq" podUID="38b8e600-068c-413b-b73c-244dd27e942c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:05:57.624308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:57.624278    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:57.640110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:57.640084    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:05:58.102452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:05:58.102423    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:06:04.371578 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:04.371543    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:06:04.386004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:04.385980    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:06:04.555516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:04.555481    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:06:04.861645 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:06:04.861611    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:06:05.247251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:05.247218    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:06:06.159498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:06.155562    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq" podUID="38b8e600-068c-413b-b73c-244dd27e942c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:06.159498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:06.158696    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz" podUID="5dc521f2-3dfa-470a-b780-ee43a2a113c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:06:08.023060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:08.023026    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"]
Oct 30 22:06:08.023812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:08.023780    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h" podUID="5f795a2a-499e-44cf-83c3-f2c34e94df56" containerName="maas-api" containerID="cri-o://e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa" gracePeriod=30
Oct 30 22:06:09.360863 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.360834    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:06:09.386257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.386172    4282 generic.go:356] "Generic (PLEG): container finished" podID="5f795a2a-499e-44cf-83c3-f2c34e94df56" containerID="e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa" exitCode=0
Oct 30 22:06:09.386257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.386235    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"
Oct 30 22:06:09.386529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.386259    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h" event={"ID":"5f795a2a-499e-44cf-83c3-f2c34e94df56","Type":"ContainerDied","Data":"e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa"}
Oct 30 22:06:09.386529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.386325    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h" event={"ID":"5f795a2a-499e-44cf-83c3-f2c34e94df56","Type":"ContainerDied","Data":"65336f1661cbb57d0473dfd5993ce25be97d8add78c18852a0a14c67443e6d5a"}
Oct 30 22:06:09.386529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.386395    4282 scope.go:122] "RemoveContainer" containerID="e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa"
Oct 30 22:06:09.399542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.399155    4282 scope.go:122] "RemoveContainer" containerID="e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa"
Oct 30 22:06:09.399542 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:06:09.399500    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa\": container with ID starting with e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa not found: ID does not exist" containerID="e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa"
Oct 30 22:06:09.399822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.399657    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa"} err="failed to get container status \"e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa\": rpc error: code = NotFound desc = could not find container \"e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa\": container with ID starting with e61fb0258ec42cf0609cfa7b5f4ba5ddc07d9b3f893586dbfe61d20f777d50aa not found: ID does not exist"
Oct 30 22:06:09.447001 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.446968    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f795a2a-499e-44cf-83c3-f2c34e94df56-kube-api-access-49d2g\" (UniqueName: \"kubernetes.io/projected/5f795a2a-499e-44cf-83c3-f2c34e94df56-kube-api-access-49d2g\") pod \"5f795a2a-499e-44cf-83c3-f2c34e94df56\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") "
Oct 30 22:06:09.447351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.447103    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f795a2a-499e-44cf-83c3-f2c34e94df56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f795a2a-499e-44cf-83c3-f2c34e94df56-openshift-service-ca\") pod \"5f795a2a-499e-44cf-83c3-f2c34e94df56\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") "
Oct 30 22:06:09.447351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.447252    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-metrics-certs\") pod \"5f795a2a-499e-44cf-83c3-f2c34e94df56\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") "
Oct 30 22:06:09.447615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.447424    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-maas-api-tls\") pod \"5f795a2a-499e-44cf-83c3-f2c34e94df56\" (UID: \"5f795a2a-499e-44cf-83c3-f2c34e94df56\") "
Oct 30 22:06:09.447930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.447905    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f795a2a-499e-44cf-83c3-f2c34e94df56-openshift-service-ca" pod "5f795a2a-499e-44cf-83c3-f2c34e94df56" (UID: "5f795a2a-499e-44cf-83c3-f2c34e94df56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:09.448283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.448257    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f795a2a-499e-44cf-83c3-f2c34e94df56-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.450367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.450336    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-maas-api-tls" pod "5f795a2a-499e-44cf-83c3-f2c34e94df56" (UID: "5f795a2a-499e-44cf-83c3-f2c34e94df56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:09.488203 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.488151    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-metrics-certs" pod "5f795a2a-499e-44cf-83c3-f2c34e94df56" (UID: "5f795a2a-499e-44cf-83c3-f2c34e94df56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:09.493618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.493586    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f795a2a-499e-44cf-83c3-f2c34e94df56-kube-api-access-49d2g" pod "5f795a2a-499e-44cf-83c3-f2c34e94df56" (UID: "5f795a2a-499e-44cf-83c3-f2c34e94df56"). InnerVolumeSpecName "kube-api-access-49d2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:09.550653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.550627    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.550653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.550652    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f795a2a-499e-44cf-83c3-f2c34e94df56-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.550832 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.550663    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49d2g\" (UniqueName: \"kubernetes.io/projected/5f795a2a-499e-44cf-83c3-f2c34e94df56-kube-api-access-49d2g\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:09.705157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.705097    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"]
Oct 30 22:06:09.709164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:09.709142    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d81a2-7c94f47db8-b9p7h"]
Oct 30 22:06:11.489340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:11.489310    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f795a2a-499e-44cf-83c3-f2c34e94df56" path="/var/lib/kubelet/pods/5f795a2a-499e-44cf-83c3-f2c34e94df56/volumes"
Oct 30 22:06:12.497620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.497583    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"]
Oct 30 22:06:12.498431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.498411    4282 cpu_manager.go:414] "containerMap: removing container" podUID="5f795a2a-499e-44cf-83c3-f2c34e94df56" containerName="maas-api"
Oct 30 22:06:12.498431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.498430    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f795a2a-499e-44cf-83c3-f2c34e94df56" containerName="maas-api"
Oct 30 22:06:12.498641 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.498603    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f795a2a-499e-44cf-83c3-f2c34e94df56" containerName="maas-api"
Oct 30 22:06:12.504019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.503593    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.511091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.510939    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f73c9\"/\"openshift-service-ca.crt\""
Oct 30 22:06:12.516388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.516367    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"]
Oct 30 22:06:12.518948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.518447    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f73c9\"/\"default-dockercfg-jmr6z\""
Oct 30 22:06:12.518948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.518580    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f73c9\"/\"kube-root-ca.crt\""
Oct 30 22:06:12.518948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.518701    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0f73c9\"/\"auth-test-model-0f73c9-kserve-self-signed-certs\""
Oct 30 22:06:12.569901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.569693    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-home\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.569901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.569763    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdcp\" (UniqueName: \"kubernetes.io/projected/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-kube-api-access-vxdcp\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.569901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.569890    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-model-cache\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.570209 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.569916    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tmp-dir\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.570396 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.570375    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tls-certs\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.570456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.570407    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-dshm\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.678757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.678722    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-home\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.750668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.750063    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-home\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.759459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.759421    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdcp\" (UniqueName: \"kubernetes.io/projected/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-kube-api-access-vxdcp\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.759642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.759609    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-model-cache\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.759695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.759658    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tmp-dir\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.759918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.759899    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tls-certs\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.759965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.759942    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-dshm\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.763163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.762079    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-model-cache\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.763163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.762739    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tmp-dir\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.783238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.783165    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tls-certs\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.860497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.853439    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdcp\" (UniqueName: \"kubernetes.io/projected/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-kube-api-access-vxdcp\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:12.874090 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:12.861015    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-dshm\") pod \"auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") " pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:13.128902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.127972    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:13.187449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.187034    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:06:13.276234 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.276197    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b8e600-068c-413b-b73c-244dd27e942c-kube-api-access-ktrbg\" (UniqueName: \"kubernetes.io/projected/38b8e600-068c-413b-b73c-244dd27e942c-kube-api-access-ktrbg\") pod \"38b8e600-068c-413b-b73c-244dd27e942c\" (UID: \"38b8e600-068c-413b-b73c-244dd27e942c\") "
Oct 30 22:06:13.276370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.276300    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b8e600-068c-413b-b73c-244dd27e942c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b8e600-068c-413b-b73c-244dd27e942c-plugins-config-volume\") pod \"38b8e600-068c-413b-b73c-244dd27e942c\" (UID: \"38b8e600-068c-413b-b73c-244dd27e942c\") "
Oct 30 22:06:13.277123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.277049    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b8e600-068c-413b-b73c-244dd27e942c-plugins-config-volume" pod "38b8e600-068c-413b-b73c-244dd27e942c" (UID: "38b8e600-068c-413b-b73c-244dd27e942c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:13.277295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.277272    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38b8e600-068c-413b-b73c-244dd27e942c-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:13.279088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.279062    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b8e600-068c-413b-b73c-244dd27e942c-kube-api-access-ktrbg" pod "38b8e600-068c-413b-b73c-244dd27e942c" (UID: "38b8e600-068c-413b-b73c-244dd27e942c"). InnerVolumeSpecName "kube-api-access-ktrbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:13.299959 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.299938    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:06:13.377963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.377892    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc521f2-3dfa-470a-b780-ee43a2a113c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc521f2-3dfa-470a-b780-ee43a2a113c4-plugins-config-volume\") pod \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\" (UID: \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\") "
Oct 30 22:06:13.377963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.377946    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc521f2-3dfa-470a-b780-ee43a2a113c4-kube-api-access-kqxcv\" (UniqueName: \"kubernetes.io/projected/5dc521f2-3dfa-470a-b780-ee43a2a113c4-kube-api-access-kqxcv\") pod \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\" (UID: \"5dc521f2-3dfa-470a-b780-ee43a2a113c4\") "
Oct 30 22:06:13.378280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.378256    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc521f2-3dfa-470a-b780-ee43a2a113c4-plugins-config-volume" pod "5dc521f2-3dfa-470a-b780-ee43a2a113c4" (UID: "5dc521f2-3dfa-470a-b780-ee43a2a113c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:13.378723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.378703    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc521f2-3dfa-470a-b780-ee43a2a113c4-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:13.378802 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.378726    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktrbg\" (UniqueName: \"kubernetes.io/projected/38b8e600-068c-413b-b73c-244dd27e942c-kube-api-access-ktrbg\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:13.380314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.380287    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc521f2-3dfa-470a-b780-ee43a2a113c4-kube-api-access-kqxcv" pod "5dc521f2-3dfa-470a-b780-ee43a2a113c4" (UID: "5dc521f2-3dfa-470a-b780-ee43a2a113c4"). InnerVolumeSpecName "kube-api-access-kqxcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:13.421368 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.421339    4282 generic.go:356] "Generic (PLEG): container finished" podID="5dc521f2-3dfa-470a-b780-ee43a2a113c4" containerID="d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60" exitCode=0
Oct 30 22:06:13.421519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.421379    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz" event={"ID":"5dc521f2-3dfa-470a-b780-ee43a2a113c4","Type":"ContainerDied","Data":"d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60"}
Oct 30 22:06:13.421519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.421392    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"
Oct 30 22:06:13.421519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.421494    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz" event={"ID":"5dc521f2-3dfa-470a-b780-ee43a2a113c4","Type":"ContainerDied","Data":"3ac5ab959ee45480863d7c111685714acf859e1ed1cda38cf7418b3ae8415abf"}
Oct 30 22:06:13.421695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.421527    4282 scope.go:122] "RemoveContainer" containerID="d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60"
Oct 30 22:06:13.423393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.423121    4282 generic.go:356] "Generic (PLEG): container finished" podID="38b8e600-068c-413b-b73c-244dd27e942c" containerID="eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28" exitCode=0
Oct 30 22:06:13.423393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.423173    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq" event={"ID":"38b8e600-068c-413b-b73c-244dd27e942c","Type":"ContainerDied","Data":"eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28"}
Oct 30 22:06:13.423654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.423630    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq" event={"ID":"38b8e600-068c-413b-b73c-244dd27e942c","Type":"ContainerDied","Data":"21b3ca17c6eb9d29adc707059484fb930e2b512ae567fd8e3d7c8d21c85179f3"}
Oct 30 22:06:13.423654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.423228    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"
Oct 30 22:06:13.439440 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.439413    4282 scope.go:122] "RemoveContainer" containerID="d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60"
Oct 30 22:06:13.439942 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:06:13.439917    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60\": container with ID starting with d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60 not found: ID does not exist" containerID="d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60"
Oct 30 22:06:13.440020 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.439956    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60"} err="failed to get container status \"d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60\": rpc error: code = NotFound desc = could not find container \"d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60\": container with ID starting with d0d5fc33b67a14f7c86238edd985bd6ffa8410e9aaafcd1a6b1cafe1089b6c60 not found: ID does not exist"
Oct 30 22:06:13.440020 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.439981    4282 scope.go:122] "RemoveContainer" containerID="eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28"
Oct 30 22:06:13.452252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.452228    4282 scope.go:122] "RemoveContainer" containerID="eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28"
Oct 30 22:06:13.452576 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:06:13.452556    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28\": container with ID starting with eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28 not found: ID does not exist" containerID="eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28"
Oct 30 22:06:13.452644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.452583    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28"} err="failed to get container status \"eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28\": rpc error: code = NotFound desc = could not find container \"eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28\": container with ID starting with eda1c1df256f727bbddf9b5ce727ff825c26b0aac57b88c89c66f7f8b120ea28 not found: ID does not exist"
Oct 30 22:06:13.463281 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:06:13.463251    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0702d6a6_7a28_4444_96e3_f53c6c7ac1e9.slice/crio-e599aec41c09d804048ebbc30bcbb57e5c62fec8a314c17832e48da469ea28d5 WatchSource:0}: Error finding container e599aec41c09d804048ebbc30bcbb57e5c62fec8a314c17832e48da469ea28d5: Status 404 returned error can't find the container with id e599aec41c09d804048ebbc30bcbb57e5c62fec8a314c17832e48da469ea28d5
Oct 30 22:06:13.467724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.467703    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"]
Oct 30 22:06:13.467841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.467732    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"]
Oct 30 22:06:13.467968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.467918    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d81a2-6dd9846759-8kczz"]
Oct 30 22:06:13.476275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.476253    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"]
Oct 30 22:06:13.480277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.480251    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqxcv\" (UniqueName: \"kubernetes.io/projected/5dc521f2-3dfa-470a-b780-ee43a2a113c4-kube-api-access-kqxcv\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:13.483479 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.483437    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d81a2-7fccd6dbdf-zsvgq"]
Oct 30 22:06:13.987577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:13.987539    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"]
Oct 30 22:06:14.443513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:14.443449    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz" event={"ID":"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9","Type":"ContainerStarted","Data":"31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651"}
Oct 30 22:06:14.443513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:14.443511    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz" event={"ID":"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9","Type":"ContainerStarted","Data":"e599aec41c09d804048ebbc30bcbb57e5c62fec8a314c17832e48da469ea28d5"}
Oct 30 22:06:14.464192 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:14.464139    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz" podStartSLOduration=2.46412451 podStartE2EDuration="2.46412451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:14.460734313 +0000 UTC m=+1593.118830537" watchObservedRunningTime="2026-10-30 22:06:14.46412451 +0000 UTC m=+1593.122220753"
Oct 30 22:06:15.137970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:15.135378    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:15.466140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:15.466104    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b8e600-068c-413b-b73c-244dd27e942c" path="/var/lib/kubelet/pods/38b8e600-068c-413b-b73c-244dd27e942c/volumes"
Oct 30 22:06:15.466580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:15.466565    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc521f2-3dfa-470a-b780-ee43a2a113c4" path="/var/lib/kubelet/pods/5dc521f2-3dfa-470a-b780-ee43a2a113c4/volumes"
Oct 30 22:06:17.468142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:17.468116    4282 generic.go:356] "Generic (PLEG): container finished" podID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerID="7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da" exitCode=0
Oct 30 22:06:17.468530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:17.468158    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" event={"ID":"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c","Type":"ContainerDied","Data":"7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da"}
Oct 30 22:06:17.468611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:17.468593    4282 scope.go:122] "RemoveContainer" containerID="7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da"
Oct 30 22:06:17.470415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:17.470371    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:06:17.470565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:17.470454    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:06:17.470565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:17.470515    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:06:18.523008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:18.522959    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" event={"ID":"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c","Type":"ContainerStarted","Data":"07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7"}
Oct 30 22:06:19.009494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:19.005864    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:06:19.009494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:19.006004    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:06:19.009494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:19.006051    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 22:06:20.001818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:20.001782    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:06:20.002232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:20.001857    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 22:06:21.001977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.001940    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:06:21.002403 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.001993    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 22:06:21.092659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.092626    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"]
Oct 30 22:06:21.094937 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.094911    4282 cpu_manager.go:414] "containerMap: removing container" podUID="5dc521f2-3dfa-470a-b780-ee43a2a113c4" containerName="payload-pre-processing"
Oct 30 22:06:21.095813 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.095794    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc521f2-3dfa-470a-b780-ee43a2a113c4" containerName="payload-pre-processing"
Oct 30 22:06:21.095963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.095948    4282 cpu_manager.go:414] "containerMap: removing container" podUID="38b8e600-068c-413b-b73c-244dd27e942c" containerName="payload-processing"
Oct 30 22:06:21.096066 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.096053    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b8e600-068c-413b-b73c-244dd27e942c" containerName="payload-processing"
Oct 30 22:06:21.096404 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.096385    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc521f2-3dfa-470a-b780-ee43a2a113c4" containerName="payload-pre-processing"
Oct 30 22:06:21.096569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.096556    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b8e600-068c-413b-b73c-244dd27e942c" containerName="payload-processing"
Oct 30 22:06:21.105571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.105551    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"]
Oct 30 22:06:21.105683 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.105654    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.108680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.108661    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0a2577\"/\"default-dockercfg-nqbwk\""
Oct 30 22:06:21.108966 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.108948    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0a2577\"/\"auth-test-model-0a2577-kserve-self-signed-certs\""
Oct 30 22:06:21.109080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.109022    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0a2577\"/\"kube-root-ca.crt\""
Oct 30 22:06:21.112319 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.112300    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0a2577\"/\"openshift-service-ca.crt\""
Oct 30 22:06:21.239777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.239735    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tls-certs\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.240066 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.240046    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-model-cache\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.240315 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.240292    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gglzr\" (UniqueName: \"kubernetes.io/projected/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-kube-api-access-gglzr\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.243152 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.243122    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-dshm\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.243382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.243363    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-home\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.243596 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.243577    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tmp-dir\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.354884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.354783    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gglzr\" (UniqueName: \"kubernetes.io/projected/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-kube-api-access-gglzr\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.355782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.355741    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-dshm\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.356484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.356441    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-home\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.357173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.357046    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-home\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.357727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.357660    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tmp-dir\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.357889 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.357871    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tmp-dir\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.367745 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.367697    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tls-certs\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.368008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.367987    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-model-cache\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.380209 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.377351    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tls-certs\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.380718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.380688    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-model-cache\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.410496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.406814    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gglzr\" (UniqueName: \"kubernetes.io/projected/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-kube-api-access-gglzr\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.427820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.427786    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-dshm\") pod \"auth-test-model-0a2577-kserve-556bdf4fb9-kfctb\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") " pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.456499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.438682    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:21.806023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:21.805994    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"]
Oct 30 22:06:21.806720 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:06:21.806686    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc185ae6_ec6b_4117_8bdb_d88f8fbf2ae4.slice/crio-ac548b41418a9006d3d20fef783976051fc95c7af65eb5e2f6c57121514fb040 WatchSource:0}: Error finding container ac548b41418a9006d3d20fef783976051fc95c7af65eb5e2f6c57121514fb040: Status 404 returned error can't find the container with id ac548b41418a9006d3d20fef783976051fc95c7af65eb5e2f6c57121514fb040
Oct 30 22:06:22.008133 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:22.008092    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:06:22.008801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:22.008759    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 22:06:22.570494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:22.569453    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb" event={"ID":"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4","Type":"ContainerStarted","Data":"c9c89cbf0e5bcbe4bbfad871645e33ca46775784434cd827d56bc9790fe55858"}
Oct 30 22:06:22.570494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:22.569529    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb" event={"ID":"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4","Type":"ContainerStarted","Data":"ac548b41418a9006d3d20fef783976051fc95c7af65eb5e2f6c57121514fb040"}
Oct 30 22:06:22.597353 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:22.597304    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb" podStartSLOduration=1.597292473 podStartE2EDuration="1.597292473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:22.597089405 +0000 UTC m=+1601.255185630" watchObservedRunningTime="2026-10-30 22:06:22.597292473 +0000 UTC m=+1601.255388696"
Oct 30 22:06:23.002192 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:23.002153    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Oct 30 22:06:23.002410 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:23.002205    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Oct 30 22:06:23.129646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:23.129615    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:23.142970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:23.142942    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:23.450491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:23.444088    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:23.596437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:23.596399    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:06:24.171088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:24.171030    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:06:24.172666 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:24.172644    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:06:24.189062 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:24.189031    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:06:29.010223 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.010178    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" containerID="cri-o://6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c" gracePeriod=45
Oct 30 22:06:29.117903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.117870    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": dial tcp 10.133.0.153:8000: connect: connection refused"
Oct 30 22:06:29.375371 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.375349    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:06:29.443955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.443920    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97475a44-715c-4ff6-8f75-a54fc074be6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/97475a44-715c-4ff6-8f75-a54fc074be6a-tls-certs\") pod \"97475a44-715c-4ff6-8f75-a54fc074be6a\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") "
Oct 30 22:06:29.444141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.444008    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-model-cache\") pod \"97475a44-715c-4ff6-8f75-a54fc074be6a\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") "
Oct 30 22:06:29.444141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.444049    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97475a44-715c-4ff6-8f75-a54fc074be6a-kube-api-access-r7l2v\" (UniqueName: \"kubernetes.io/projected/97475a44-715c-4ff6-8f75-a54fc074be6a-kube-api-access-r7l2v\") pod \"97475a44-715c-4ff6-8f75-a54fc074be6a\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") "
Oct 30 22:06:29.444286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.444139    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-dshm\") pod \"97475a44-715c-4ff6-8f75-a54fc074be6a\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") "
Oct 30 22:06:29.444286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.444199    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-home\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-home\") pod \"97475a44-715c-4ff6-8f75-a54fc074be6a\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") "
Oct 30 22:06:29.444391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.444327    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-model-cache" pod "97475a44-715c-4ff6-8f75-a54fc074be6a" (UID: "97475a44-715c-4ff6-8f75-a54fc074be6a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:29.444391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.444358    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-tmp-dir\") pod \"97475a44-715c-4ff6-8f75-a54fc074be6a\" (UID: \"97475a44-715c-4ff6-8f75-a54fc074be6a\") "
Oct 30 22:06:29.445151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.445123    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-home" pod "97475a44-715c-4ff6-8f75-a54fc074be6a" (UID: "97475a44-715c-4ff6-8f75-a54fc074be6a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:29.445315 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.445146    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:29.445590 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.445564    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-tmp-dir" pod "97475a44-715c-4ff6-8f75-a54fc074be6a" (UID: "97475a44-715c-4ff6-8f75-a54fc074be6a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:29.447079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.447056    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97475a44-715c-4ff6-8f75-a54fc074be6a-kube-api-access-r7l2v" pod "97475a44-715c-4ff6-8f75-a54fc074be6a" (UID: "97475a44-715c-4ff6-8f75-a54fc074be6a"). InnerVolumeSpecName "kube-api-access-r7l2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:29.451459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.451434    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97475a44-715c-4ff6-8f75-a54fc074be6a-tls-certs" pod "97475a44-715c-4ff6-8f75-a54fc074be6a" (UID: "97475a44-715c-4ff6-8f75-a54fc074be6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:29.451595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.451536    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-dshm" pod "97475a44-715c-4ff6-8f75-a54fc074be6a" (UID: "97475a44-715c-4ff6-8f75-a54fc074be6a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:29.547549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.547452    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7l2v\" (UniqueName: \"kubernetes.io/projected/97475a44-715c-4ff6-8f75-a54fc074be6a-kube-api-access-r7l2v\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:29.547549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.547500    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:29.547549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.547516    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:29.547549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.547527    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97475a44-715c-4ff6-8f75-a54fc074be6a-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:29.547549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.547536    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97475a44-715c-4ff6-8f75-a54fc074be6a-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:29.679884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.679855    4282 generic.go:356] "Generic (PLEG): container finished" podID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerID="6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c" exitCode=0
Oct 30 22:06:29.680055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.679896    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"
Oct 30 22:06:29.680055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.679949    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" event={"ID":"97475a44-715c-4ff6-8f75-a54fc074be6a","Type":"ContainerDied","Data":"6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c"}
Oct 30 22:06:29.680055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.679990    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw" event={"ID":"97475a44-715c-4ff6-8f75-a54fc074be6a","Type":"ContainerDied","Data":"f3f247c05bfeed2636267eeb2944fde87ab116a07e0c168f3cad674e8e155602"}
Oct 30 22:06:29.680055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.680011    4282 scope.go:122] "RemoveContainer" containerID="6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c"
Oct 30 22:06:29.691048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.691029    4282 scope.go:122] "RemoveContainer" containerID="6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c"
Oct 30 22:06:29.691370 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:06:29.691341    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c\": container with ID starting with 6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c not found: ID does not exist" containerID="6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c"
Oct 30 22:06:29.691498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.691382    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c"} err="failed to get container status \"6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c\": rpc error: code = NotFound desc = could not find container \"6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c\": container with ID starting with 6f7e52b77b905833930342863c726b8439528a8fe4874042760848533ef2689c not found: ID does not exist"
Oct 30 22:06:29.698294 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.698266    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"]
Oct 30 22:06:29.702769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:29.702752    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-5d81a2/praxis-route-5d81a2-kserve-bbf66979f-zk8vw"]
Oct 30 22:06:31.439172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:31.439144    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:31.452103 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:31.452080    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:31.465688 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:31.465661    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" path="/var/lib/kubelet/pods/97475a44-715c-4ff6-8f75-a54fc074be6a/volumes"
Oct 30 22:06:31.707675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:31.707598    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:06:34.650728 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:06:34.650691    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:06:40.346833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.346772    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"]
Oct 30 22:06:40.370644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.370536    4282 cpu_manager.go:414] "containerMap: removing container" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main"
Oct 30 22:06:40.370644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.370565    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main"
Oct 30 22:06:40.370934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.370894    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="97475a44-715c-4ff6-8f75-a54fc074be6a" containerName="main"
Oct 30 22:06:40.417498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.415717    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"]
Oct 30 22:06:40.417498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.415882    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.434498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.427400    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-18ba2f-openshift-default-dockercfg-6d5d5\""
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.566888    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-envoy\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.566963    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-istio-token\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.567102    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-data\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.567148    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746fa886-2c5e-4eca-98f9-734076279325-istio-podinfo\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.567217    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqm8z\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-kube-api-access-qqm8z\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.567323    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.567488    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746fa886-2c5e-4eca-98f9-734076279325-istiod-ca-cert\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.567640    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-credential-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.583515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.567716    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-certs\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.690547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.690483    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746fa886-2c5e-4eca-98f9-734076279325-istiod-ca-cert\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.690763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.690629    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-credential-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.690763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.690695    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-certs\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.690913 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.690780    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-envoy\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.690913 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.690825    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-istio-token\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.691027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.690968    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-data\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.691087 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.691023    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746fa886-2c5e-4eca-98f9-734076279325-istio-podinfo\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.691087 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.691077    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqm8z\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-kube-api-access-qqm8z\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.691203 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.691181    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.691850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.691825    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.692313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.692292    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746fa886-2c5e-4eca-98f9-734076279325-istiod-ca-cert\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.692521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.692501    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-credential-socket\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.692790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.692769    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-certs\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.696161 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.695724    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-envoy\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.699875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.696365    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-data\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.703235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.703191    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746fa886-2c5e-4eca-98f9-734076279325-istio-podinfo\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.710831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.709021    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-istio-token\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.722457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.722010    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqm8z\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-kube-api-access-qqm8z\") pod \"e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") " pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:40.807932 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:40.807894    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:41.183445 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:06:41.183411    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746fa886_2c5e_4eca_98f9_734076279325.slice/crio-ac871aa10d100397ae058e29179951a4b8a7374c0ba9d4873b2baad90d9e2322 WatchSource:0}: Error finding container ac871aa10d100397ae058e29179951a4b8a7374c0ba9d4873b2baad90d9e2322: Status 404 returned error can't find the container with id ac871aa10d100397ae058e29179951a4b8a7374c0ba9d4873b2baad90d9e2322
Oct 30 22:06:41.186965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:41.186912    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:06:41.187240 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:41.187006    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:06:41.187240 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:41.187109    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:06:41.199638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:41.199607    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"]
Oct 30 22:06:41.889533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:41.889495    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" event={"ID":"746fa886-2c5e-4eca-98f9-734076279325","Type":"ContainerStarted","Data":"d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8"}
Oct 30 22:06:41.889954 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:41.889559    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" event={"ID":"746fa886-2c5e-4eca-98f9-734076279325","Type":"ContainerStarted","Data":"ac871aa10d100397ae058e29179951a4b8a7374c0ba9d4873b2baad90d9e2322"}
Oct 30 22:06:41.915442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:41.915387    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" podStartSLOduration=1.915373623 podStartE2EDuration="1.915373623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:41.912317077 +0000 UTC m=+1620.570413301" watchObservedRunningTime="2026-10-30 22:06:41.915373623 +0000 UTC m=+1620.573469908"
Oct 30 22:06:42.809037 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:42.809003    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:42.810154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:42.810134    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:06:42.810236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:42.810194    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 22:06:43.809263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:43.809225    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:06:43.809714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:43.809288    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 22:06:44.809095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:44.809061    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:06:44.809287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:44.809113    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 22:06:45.808745 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:45.808708    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:06:45.808941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:45.808759    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 22:06:46.808875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:46.808841    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 30 22:06:46.809339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:46.808941    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 30 22:06:47.814085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:47.814055    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:47.814546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:47.814527    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:47.815366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:47.815345    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:06:59.104824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:59.104790    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"]
Oct 30 22:06:59.105414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:06:59.105043    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy" containerID="cri-o://94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e" gracePeriod=30
Oct 30 22:07:04.413620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.413593    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:07:04.600837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.600809    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-token\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-token\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.600929    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-credential-socket\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601023    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-podinfo\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601084    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istiod-ca-cert\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601122    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-envoy\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601244    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-credential-socket" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:04.601333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601255    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-certs\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601328    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-data\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601451 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601430    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-kube-api-access-7qqn4\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-kube-api-access-7qqn4\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601518    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-socket\") pod \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\" (UID: \"b3ebeb51-11ed-4109-bf1f-5281e08a6445\") "
Oct 30 22:07:04.601609 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601535    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-certs" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:04.601609 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601580    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istiod-ca-cert" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:04.601810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.601791    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-data" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:04.602034 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.602006    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-socket" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:04.602330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.602297    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.602330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.602330    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.602552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.602345    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.602552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.602358    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.602552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.602372    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.603378 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.603357    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-token" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:04.603458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.603445    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-podinfo" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:04.603926 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.603905    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-kube-api-access-7qqn4" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "kube-api-access-7qqn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:04.604197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.604180    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-envoy" pod "b3ebeb51-11ed-4109-bf1f-5281e08a6445" (UID: "b3ebeb51-11ed-4109-bf1f-5281e08a6445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:04.708497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.706029    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.708497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.706095    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.708497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.706113    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3ebeb51-11ed-4109-bf1f-5281e08a6445-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.708497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:04.706125    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qqn4\" (UniqueName: \"kubernetes.io/projected/b3ebeb51-11ed-4109-bf1f-5281e08a6445-kube-api-access-7qqn4\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:04.767363 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:07:04.767322    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:07:05.190100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.190067    4282 generic.go:356] "Generic (PLEG): container finished" podID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerID="94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e" exitCode=0
Oct 30 22:07:05.190273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.190113    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"
Oct 30 22:07:05.190273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.190152    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" event={"ID":"b3ebeb51-11ed-4109-bf1f-5281e08a6445","Type":"ContainerDied","Data":"94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e"}
Oct 30 22:07:05.190273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.190205    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p" event={"ID":"b3ebeb51-11ed-4109-bf1f-5281e08a6445","Type":"ContainerDied","Data":"e14b8b3226d0dafec407adb92d58d331d5a23e0842191d069e96f832bb6235d9"}
Oct 30 22:07:05.190273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.190228    4282 scope.go:122] "RemoveContainer" containerID="94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e"
Oct 30 22:07:05.201790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.201772    4282 scope.go:122] "RemoveContainer" containerID="94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e"
Oct 30 22:07:05.202054 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:07:05.202032    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e\": container with ID starting with 94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e not found: ID does not exist" containerID="94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e"
Oct 30 22:07:05.202112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.202067    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e"} err="failed to get container status \"94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e\": rpc error: code = NotFound desc = could not find container \"94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e\": container with ID starting with 94f3969353592d63ceb2ab3b21dee3e17a79157abc4f92c3d750b05ac564482e not found: ID does not exist"
Oct 30 22:07:05.213691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.213666    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"]
Oct 30 22:07:05.218755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.218734    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d81a2-openshift-default-67b8c9ffc-4sc9p"]
Oct 30 22:07:05.510456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:05.510369    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" path="/var/lib/kubelet/pods/b3ebeb51-11ed-4109-bf1f-5281e08a6445/volumes"
Oct 30 22:07:09.802612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:09.802581    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"]
Oct 30 22:07:12.026742 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:12.026703    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:07:14.803266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:14.803228    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl" podUID="dcbe436f-d58a-45d8-a33c-23d32e87f673" containerName="payload-processing" containerID="cri-o://c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6" gracePeriod=25
Oct 30 22:07:15.052969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.052935    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"]
Oct 30 22:07:15.053339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.053185    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9" podUID="f9b2d328-8340-4a8c-ace4-f1525004c794" containerName="maas-api" containerID="cri-o://47327a57bb8d0158054070eff1c9afc3cfca6f8d39fff7cb0a77df372c884336" gracePeriod=30
Oct 30 22:07:15.192397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.192369    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:07:15.225837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.225805    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbe436f-d58a-45d8-a33c-23d32e87f673-kube-api-access-qmm79\" (UniqueName: \"kubernetes.io/projected/dcbe436f-d58a-45d8-a33c-23d32e87f673-kube-api-access-qmm79\") pod \"dcbe436f-d58a-45d8-a33c-23d32e87f673\" (UID: \"dcbe436f-d58a-45d8-a33c-23d32e87f673\") "
Oct 30 22:07:15.226160 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.226137    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbe436f-d58a-45d8-a33c-23d32e87f673-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe436f-d58a-45d8-a33c-23d32e87f673-plugins-config-volume\") pod \"dcbe436f-d58a-45d8-a33c-23d32e87f673\" (UID: \"dcbe436f-d58a-45d8-a33c-23d32e87f673\") "
Oct 30 22:07:15.226592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.226560    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbe436f-d58a-45d8-a33c-23d32e87f673-plugins-config-volume" pod "dcbe436f-d58a-45d8-a33c-23d32e87f673" (UID: "dcbe436f-d58a-45d8-a33c-23d32e87f673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:15.227762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.227735    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcbe436f-d58a-45d8-a33c-23d32e87f673-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.228771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.228748    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbe436f-d58a-45d8-a33c-23d32e87f673-kube-api-access-qmm79" pod "dcbe436f-d58a-45d8-a33c-23d32e87f673" (UID: "dcbe436f-d58a-45d8-a33c-23d32e87f673"). InnerVolumeSpecName "kube-api-access-qmm79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:15.539401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.538553    4282 generic.go:356] "Generic (PLEG): container finished" podID="dcbe436f-d58a-45d8-a33c-23d32e87f673" containerID="c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6" exitCode=0
Oct 30 22:07:15.539401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.538804    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"
Oct 30 22:07:15.543552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.543523    4282 generic.go:356] "Generic (PLEG): container finished" podID="f9b2d328-8340-4a8c-ace4-f1525004c794" containerID="47327a57bb8d0158054070eff1c9afc3cfca6f8d39fff7cb0a77df372c884336" exitCode=0
Oct 30 22:07:15.627288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.574434    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmm79\" (UniqueName: \"kubernetes.io/projected/dcbe436f-d58a-45d8-a33c-23d32e87f673-kube-api-access-qmm79\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.735974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.735919    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl" event={"ID":"dcbe436f-d58a-45d8-a33c-23d32e87f673","Type":"ContainerDied","Data":"c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6"}
Oct 30 22:07:15.737081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.737042    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl" event={"ID":"dcbe436f-d58a-45d8-a33c-23d32e87f673","Type":"ContainerDied","Data":"f00a64636ef3d55563062f86534b7cfd919b7370b2509978013929601a34bb5d"}
Oct 30 22:07:15.737482 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.737399    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9" event={"ID":"f9b2d328-8340-4a8c-ace4-f1525004c794","Type":"ContainerDied","Data":"47327a57bb8d0158054070eff1c9afc3cfca6f8d39fff7cb0a77df372c884336"}
Oct 30 22:07:15.741069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.737719    4282 scope.go:122] "RemoveContainer" containerID="c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6"
Oct 30 22:07:15.741328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.739899    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:07:15.777697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.775670    4282 scope.go:122] "RemoveContainer" containerID="c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6"
Oct 30 22:07:15.794601 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:07:15.792054    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6\": container with ID starting with c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6 not found: ID does not exist" containerID="c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6"
Oct 30 22:07:15.794601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.792103    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6"} err="failed to get container status \"c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6\": rpc error: code = NotFound desc = could not find container \"c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6\": container with ID starting with c289f3a6e11287698175671c0aebea8fc0256428970855eeaec6bd076b97cdf6 not found: ID does not exist"
Oct 30 22:07:15.802494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.801414    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-metrics-certs\") pod \"f9b2d328-8340-4a8c-ace4-f1525004c794\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") "
Oct 30 22:07:15.802494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.801623    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b2d328-8340-4a8c-ace4-f1525004c794-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2d328-8340-4a8c-ace4-f1525004c794-openshift-service-ca\") pod \"f9b2d328-8340-4a8c-ace4-f1525004c794\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") "
Oct 30 22:07:15.802494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.801740    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b2d328-8340-4a8c-ace4-f1525004c794-kube-api-access-8xg8n\" (UniqueName: \"kubernetes.io/projected/f9b2d328-8340-4a8c-ace4-f1525004c794-kube-api-access-8xg8n\") pod \"f9b2d328-8340-4a8c-ace4-f1525004c794\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") "
Oct 30 22:07:15.802494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.801950    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-maas-api-tls\") pod \"f9b2d328-8340-4a8c-ace4-f1525004c794\" (UID: \"f9b2d328-8340-4a8c-ace4-f1525004c794\") "
Oct 30 22:07:15.802494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.802138    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b2d328-8340-4a8c-ace4-f1525004c794-openshift-service-ca" pod "f9b2d328-8340-4a8c-ace4-f1525004c794" (UID: "f9b2d328-8340-4a8c-ace4-f1525004c794"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:15.807603 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.807305    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2d328-8340-4a8c-ace4-f1525004c794-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.837612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.837542    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"]
Oct 30 22:07:15.843709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.843536    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-metrics-certs" pod "f9b2d328-8340-4a8c-ace4-f1525004c794" (UID: "f9b2d328-8340-4a8c-ace4-f1525004c794"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:15.843709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.843666    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b2d328-8340-4a8c-ace4-f1525004c794-kube-api-access-8xg8n" pod "f9b2d328-8340-4a8c-ace4-f1525004c794" (UID: "f9b2d328-8340-4a8c-ace4-f1525004c794"). InnerVolumeSpecName "kube-api-access-8xg8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:15.844014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.843994    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-maas-api-tls" pod "f9b2d328-8340-4a8c-ace4-f1525004c794" (UID: "f9b2d328-8340-4a8c-ace4-f1525004c794"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:15.844818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.844717    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2cfa71-5c4869df89-gn7sl"]
Oct 30 22:07:15.909076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.909044    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.909076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.909076    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xg8n\" (UniqueName: \"kubernetes.io/projected/f9b2d328-8340-4a8c-ace4-f1525004c794-kube-api-access-8xg8n\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:15.909316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:15.909087    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2d328-8340-4a8c-ace4-f1525004c794-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:16.613788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:16.613759    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9" event={"ID":"f9b2d328-8340-4a8c-ace4-f1525004c794","Type":"ContainerDied","Data":"27226e588a71bdeb0a483ea01fba303c0e7b889edc6dbf4c3e572d7f265386c7"}
Oct 30 22:07:16.614029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:16.614010    4282 scope.go:122] "RemoveContainer" containerID="47327a57bb8d0158054070eff1c9afc3cfca6f8d39fff7cb0a77df372c884336"
Oct 30 22:07:16.615691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:16.615666    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"
Oct 30 22:07:16.645076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:16.645052    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"]
Oct 30 22:07:16.648964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:16.648942    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2cfa71-5b5d8d5ffb-grtg9"]
Oct 30 22:07:17.467431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:17.467399    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbe436f-d58a-45d8-a33c-23d32e87f673" path="/var/lib/kubelet/pods/dcbe436f-d58a-45d8-a33c-23d32e87f673/volumes"
Oct 30 22:07:17.467936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:17.467914    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b2d328-8340-4a8c-ace4-f1525004c794" path="/var/lib/kubelet/pods/f9b2d328-8340-4a8c-ace4-f1525004c794/volumes"
Oct 30 22:07:22.824680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:22.824642    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"]
Oct 30 22:07:22.825343 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:22.824956    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy" containerID="cri-o://a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0" gracePeriod=30
Oct 30 22:07:28.208717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.208690    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:07:28.265634 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.265606    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-data\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.265842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.265654    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-istio-token\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-istio-token\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.265842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.265760    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/822606a9-93cc-468f-99d6-6a040df39808-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822606a9-93cc-468f-99d6-6a040df39808-istio-podinfo\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.265984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.265963    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-data" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:28.266047 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.266022    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-envoy\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.266169 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.266144    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-kube-api-access-gsqfg\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-kube-api-access-gsqfg\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.266340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.266321    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-certs\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.266421 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.266378    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-credential-socket\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.266421 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.266411    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822606a9-93cc-468f-99d6-6a040df39808-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822606a9-93cc-468f-99d6-6a040df39808-istiod-ca-cert\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.266734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.266501    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-socket\") pod \"822606a9-93cc-468f-99d6-6a040df39808\" (UID: \"822606a9-93cc-468f-99d6-6a040df39808\") "
Oct 30 22:07:28.267015 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.266981    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822606a9-93cc-468f-99d6-6a040df39808-istiod-ca-cert" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:28.267225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.267202    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-socket" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:28.267416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.267394    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-certs" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:28.267517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.267402    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822606a9-93cc-468f-99d6-6a040df39808-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.267517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.267440    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.267517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.267456    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.267689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.267632    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-credential-socket" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:28.268718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.268693    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-istio-token" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:28.268829 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.268796    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-envoy" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:28.269413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.269387    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/822606a9-93cc-468f-99d6-6a040df39808-istio-podinfo" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:28.269915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.269879    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-kube-api-access-gsqfg" pod "822606a9-93cc-468f-99d6-6a040df39808" (UID: "822606a9-93cc-468f-99d6-6a040df39808"). InnerVolumeSpecName "kube-api-access-gsqfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:28.368444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.368366    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.368444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.368396    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.368444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.368409    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822606a9-93cc-468f-99d6-6a040df39808-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.368444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.368422    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.368444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.368435    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsqfg\" (UniqueName: \"kubernetes.io/projected/822606a9-93cc-468f-99d6-6a040df39808-kube-api-access-gsqfg\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.368743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.368450    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822606a9-93cc-468f-99d6-6a040df39808-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:28.848501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.846391    4282 generic.go:356] "Generic (PLEG): container finished" podID="822606a9-93cc-468f-99d6-6a040df39808" containerID="a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0" exitCode=0
Oct 30 22:07:28.848501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.846642    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" event={"ID":"822606a9-93cc-468f-99d6-6a040df39808","Type":"ContainerDied","Data":"a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0"}
Oct 30 22:07:28.848501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.846696    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7" event={"ID":"822606a9-93cc-468f-99d6-6a040df39808","Type":"ContainerDied","Data":"4ca1188453c3a36c147c02af60ba8d47cdaf12fa338b9d0d063daf113461fb3b"}
Oct 30 22:07:28.848501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.846723    4282 scope.go:122] "RemoveContainer" containerID="a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0"
Oct 30 22:07:28.848501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.846931    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"
Oct 30 22:07:28.886736 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.886698    4282 scope.go:122] "RemoveContainer" containerID="37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362"
Oct 30 22:07:28.897911 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.897889    4282 scope.go:122] "RemoveContainer" containerID="a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0"
Oct 30 22:07:28.898178 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:07:28.898156    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0\": container with ID starting with a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0 not found: ID does not exist" containerID="a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0"
Oct 30 22:07:28.898231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.898188    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0"} err="failed to get container status \"a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0\": rpc error: code = NotFound desc = could not find container \"a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0\": container with ID starting with a19fb85c549a2e79839c9cd0f327cf71eb8821d0bcedfa931ee99824b2f0e5e0 not found: ID does not exist"
Oct 30 22:07:28.898231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.898207    4282 scope.go:122] "RemoveContainer" containerID="37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362"
Oct 30 22:07:28.898596 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:07:28.898563    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362\": container with ID starting with 37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362 not found: ID does not exist" containerID="37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362"
Oct 30 22:07:28.898693 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.898601    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362"} err="failed to get container status \"37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362\": rpc error: code = NotFound desc = could not find container \"37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362\": container with ID starting with 37eeca18015df06239179dde43f3b54e606b2e6026f723248e9d78d8b5e69362 not found: ID does not exist"
Oct 30 22:07:28.901935 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.901914    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"]
Oct 30 22:07:28.906876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:28.906855    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2cfa71-openshift-default-7b55b6b49-tvdt7"]
Oct 30 22:07:29.497034 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:29.496958    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822606a9-93cc-468f-99d6-6a040df39808" path="/var/lib/kubelet/pods/822606a9-93cc-468f-99d6-6a040df39808/volumes"
Oct 30 22:07:34.650312 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:07:34.650278    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:07:48.250304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.250269    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"]
Oct 30 22:07:48.251195 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251173    4282 cpu_manager.go:414] "containerMap: removing container" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251197    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251216    4282 cpu_manager.go:414] "containerMap: removing container" podUID="dcbe436f-d58a-45d8-a33c-23d32e87f673" containerName="payload-processing"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251224    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbe436f-d58a-45d8-a33c-23d32e87f673" containerName="payload-processing"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251243    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251251    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251275    4282 cpu_manager.go:414] "containerMap: removing container" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251282    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251292    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f9b2d328-8340-4a8c-ace4-f1525004c794" containerName="maas-api"
Oct 30 22:07:48.251376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251298    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b2d328-8340-4a8c-ace4-f1525004c794" containerName="maas-api"
Oct 30 22:07:48.251920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251514    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy"
Oct 30 22:07:48.251920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251530    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbe436f-d58a-45d8-a33c-23d32e87f673" containerName="payload-processing"
Oct 30 22:07:48.251920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251542    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3ebeb51-11ed-4109-bf1f-5281e08a6445" containerName="istio-proxy"
Oct 30 22:07:48.251920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.251554    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b2d328-8340-4a8c-ace4-f1525004c794" containerName="maas-api"
Oct 30 22:07:48.256947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.256928    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.259850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.259829    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-0f1f9c-openshift-default-dockercfg-dgg79\""
Oct 30 22:07:48.270111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.270089    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"]
Oct 30 22:07:48.395153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395121    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9fc\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-kube-api-access-5k9fc\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.395424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395200    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-envoy\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.395424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395331    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-socket\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.395424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395387    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-certs\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.396141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395518    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f841aa6-c794-4988-ac80-934f5106059d-istiod-ca-cert\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.396141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395584    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-data\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.396141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395685    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f841aa6-c794-4988-ac80-934f5106059d-istio-podinfo\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.396141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395825    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-credential-socket\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.396141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.395956    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-istio-token\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.502879    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9fc\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-kube-api-access-5k9fc\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.502968    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-envoy\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.503060    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-socket\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.503115    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-certs\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.503189    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f841aa6-c794-4988-ac80-934f5106059d-istiod-ca-cert\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.503252    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-data\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.503349    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f841aa6-c794-4988-ac80-934f5106059d-istio-podinfo\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.503399    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-credential-socket\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.506540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.503456    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-istio-token\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.545653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.538230    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9fc\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-kube-api-access-5k9fc\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.546629 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.546547    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-data\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.546902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.546875    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-certs\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.547129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.547107    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-socket\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.548345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.548314    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-credential-socket\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.548855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.548834    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f841aa6-c794-4988-ac80-934f5106059d-istiod-ca-cert\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.570558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.570109    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-istio-token\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.570790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.570407    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-envoy\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.596135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.596091    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f841aa6-c794-4988-ac80-934f5106059d-istio-podinfo\") pod \"e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") " pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:48.874109 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:48.874031    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:49.256082 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:49.256032    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"]
Oct 30 22:07:49.258926 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:07:49.258895    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f841aa6_c794_4988_ac80_934f5106059d.slice/crio-5a8a6bb69134e104ca55f595d4990cc9f3c319996346e7d7f8d91c8ed20b6a83 WatchSource:0}: Error finding container 5a8a6bb69134e104ca55f595d4990cc9f3c319996346e7d7f8d91c8ed20b6a83: Status 404 returned error can't find the container with id 5a8a6bb69134e104ca55f595d4990cc9f3c319996346e7d7f8d91c8ed20b6a83
Oct 30 22:07:49.261667 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:49.261629    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:07:49.261744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:49.261706    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:07:49.261744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:49.261737    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:07:50.105618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:50.105582    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" event={"ID":"5f841aa6-c794-4988-ac80-934f5106059d","Type":"ContainerStarted","Data":"983fdd40a3483427a7f6e20bb1065248cb64deb3b3691deaa858262f16f70315"}
Oct 30 22:07:50.105838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:50.105635    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" event={"ID":"5f841aa6-c794-4988-ac80-934f5106059d","Type":"ContainerStarted","Data":"5a8a6bb69134e104ca55f595d4990cc9f3c319996346e7d7f8d91c8ed20b6a83"}
Oct 30 22:07:50.130129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:50.130079    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" podStartSLOduration=2.130063228 podStartE2EDuration="2.130063228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:50.126819859 +0000 UTC m=+1688.784916081" watchObservedRunningTime="2026-10-30 22:07:50.130063228 +0000 UTC m=+1688.788159451"
Oct 30 22:07:50.874415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:50.874380    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:50.880373 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:50.880347    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:51.112702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:51.112671    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:51.113487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:51.113451    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:07:53.795216 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:53.795180    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"]
Oct 30 22:07:53.796491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:53.796451    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="822606a9-93cc-468f-99d6-6a040df39808" containerName="istio-proxy"
Oct 30 22:07:53.800655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:53.800635    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:53.809281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:53.809255    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"]
Oct 30 22:07:53.942496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:53.920706    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0f1f9c-dockercfg-kd8zs\""
Oct 30 22:07:54.050298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.026199    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c8e59-6271-46db-9028-11c4c575f169-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv\" (UID: \"418c8e59-6271-46db-9028-11c4c575f169\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:54.050298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.047991    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbkl\" (UniqueName: \"kubernetes.io/projected/418c8e59-6271-46db-9028-11c4c575f169-kube-api-access-kfbkl\") pod \"payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv\" (UID: \"418c8e59-6271-46db-9028-11c4c575f169\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:54.050298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.047680    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0f1f9c\""
Oct 30 22:07:54.152184 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.152145    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c8e59-6271-46db-9028-11c4c575f169-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv\" (UID: \"418c8e59-6271-46db-9028-11c4c575f169\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:54.242505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.227943    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbkl\" (UniqueName: \"kubernetes.io/projected/418c8e59-6271-46db-9028-11c4c575f169-kube-api-access-kfbkl\") pod \"payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv\" (UID: \"418c8e59-6271-46db-9028-11c4c575f169\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:54.242505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.159096    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c8e59-6271-46db-9028-11c4c575f169-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv\" (UID: \"418c8e59-6271-46db-9028-11c4c575f169\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:54.271002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.266733    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbkl\" (UniqueName: \"kubernetes.io/projected/418c8e59-6271-46db-9028-11c4c575f169-kube-api-access-kfbkl\") pod \"payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv\" (UID: \"418c8e59-6271-46db-9028-11c4c575f169\") " pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:54.421373 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.421341    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:54.713271 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:07:54.713242    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418c8e59_6271_46db_9028_11c4c575f169.slice/crio-c2f98e7eac388fc95a93fe6cb8f68154999e86cfcaf60b17d3f2c08501d8e26c WatchSource:0}: Error finding container c2f98e7eac388fc95a93fe6cb8f68154999e86cfcaf60b17d3f2c08501d8e26c: Status 404 returned error can't find the container with id c2f98e7eac388fc95a93fe6cb8f68154999e86cfcaf60b17d3f2c08501d8e26c
Oct 30 22:07:54.713446 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:54.713423    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"]
Oct 30 22:07:55.263385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:55.263339    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv" event={"ID":"418c8e59-6271-46db-9028-11c4c575f169","Type":"ContainerStarted","Data":"1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5"}
Oct 30 22:07:55.263876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:55.263394    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv" event={"ID":"418c8e59-6271-46db-9028-11c4c575f169","Type":"ContainerStarted","Data":"c2f98e7eac388fc95a93fe6cb8f68154999e86cfcaf60b17d3f2c08501d8e26c"}
Oct 30 22:07:55.263876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:55.263426    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:07:55.280013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:55.279967    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv" podStartSLOduration=2.279955426 podStartE2EDuration="2.279955426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:55.279320061 +0000 UTC m=+1693.937416283" watchObservedRunningTime="2026-10-30 22:07:55.279955426 +0000 UTC m=+1693.938051649"
Oct 30 22:07:57.861878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.861833    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"]
Oct 30 22:07:57.867781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.867758    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.882636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.882609    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"]
Oct 30 22:07:57.909238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909202    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.909489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909310    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.909489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909362    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.909616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909574    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.909669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909618    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.909728 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909680    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27jt\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-kube-api-access-l27jt\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.909834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909816    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.909890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909864    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:57.909946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:57.909917    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.011840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.011798    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.011867    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.011952    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.011969    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.011987    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.012104    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.012140    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.012209    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27jt\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-kube-api-access-l27jt\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.012348    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.012371    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.012403    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.012463    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.012887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.012862    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.013147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.013128    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.014389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.014362    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.014522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.014509    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.023294 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.023272    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.023615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.023596    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27jt\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-kube-api-access-l27jt\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.184622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.184593    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:07:58.486936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.486905    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"]
Oct 30 22:07:58.487526 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:07:58.487461    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaded6eba_1991_4d9d_8e01_5cc2d7a0c74e.slice/crio-0abb8aa2872b276334aa5b1a480669a3d27d269267819ca8b3ae895e22cc4228 WatchSource:0}: Error finding container 0abb8aa2872b276334aa5b1a480669a3d27d269267819ca8b3ae895e22cc4228: Status 404 returned error can't find the container with id 0abb8aa2872b276334aa5b1a480669a3d27d269267819ca8b3ae895e22cc4228
Oct 30 22:07:58.490709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.490673    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:07:58.490803 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.490745    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:07:58.490803 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:58.490773    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:07:59.295307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:59.295275    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" event={"ID":"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e","Type":"ContainerStarted","Data":"b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee"}
Oct 30 22:07:59.295852 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:59.295324    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" event={"ID":"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e","Type":"ContainerStarted","Data":"0abb8aa2872b276334aa5b1a480669a3d27d269267819ca8b3ae895e22cc4228"}
Oct 30 22:07:59.319750 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:07:59.319699    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podStartSLOduration=2.319685149 podStartE2EDuration="2.319685149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:59.317332977 +0000 UTC m=+1697.975429199" watchObservedRunningTime="2026-10-30 22:07:59.319685149 +0000 UTC m=+1697.977781371"
Oct 30 22:08:00.185138 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:00.185104    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:08:00.186849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:00.186824    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:08:00.187017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:00.186877    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 22:08:01.190497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:01.187935    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:08:01.190497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:01.188002    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 22:08:02.215495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:02.209763    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:08:02.215495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:02.209822    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 22:08:03.185002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:03.184964    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:08:03.185185 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:03.185016    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 22:08:03.431608 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:03.431576    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:08:04.185433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:04.185394    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:08:04.185665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:04.185501    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 22:08:04.848787 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:04.848720    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:08:05.193536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:05.193052    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:08:05.193536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:05.193126    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 22:08:06.185644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:06.185599    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:08:06.186177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:06.185667    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 22:08:06.273312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:06.273282    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:08:08.234628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:08.234574    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:08:08.235116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:08.234653    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:08:08.439624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:08.439591    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:08:08.440137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:08.440114    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:08:08.441342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:08.441316    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:08:20.304486 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.304436    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"]
Oct 30 22:08:20.310253 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.310219    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.317853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.317285    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0f1f9c\"/\"openshift-service-ca.crt\""
Oct 30 22:08:20.318016 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.317996    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0f1f9c\"/\"default-dockercfg-742hb\""
Oct 30 22:08:20.318165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.318149    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0f1f9c\"/\"kube-root-ca.crt\""
Oct 30 22:08:20.319165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.318353    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0f1f9c\"/\"explicit-model-0f1f9c-kserve-self-signed-certs\""
Oct 30 22:08:20.322295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.322271    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"]
Oct 30 22:08:20.346158 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.346124    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.346366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.346346    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5p2n\" (UniqueName: \"kubernetes.io/projected/58d9e761-6ac8-43f3-9b01-a65f033493c1-kube-api-access-l5p2n\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.346578 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.346555    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-model-cache\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.346705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.346598    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-tmp-dir\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.346705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.346693    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-home\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.347085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.347058    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-dshm\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.448772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.448741    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-home\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.449031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.449012    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-dshm\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.449134 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.449119    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.449177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.449153    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5p2n\" (UniqueName: \"kubernetes.io/projected/58d9e761-6ac8-43f3-9b01-a65f033493c1-kube-api-access-l5p2n\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.449233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.449182    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-home\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.449290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.449277    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-model-cache\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.449344 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.449310    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-tmp-dir\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.449758 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.449730    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-tmp-dir\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.450068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.449846    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-model-cache\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.451537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.451512    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-dshm\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.451838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.451820    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.460692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.460669    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5p2n\" (UniqueName: \"kubernetes.io/projected/58d9e761-6ac8-43f3-9b01-a65f033493c1-kube-api-access-l5p2n\") pod \"explicit-model-0f1f9c-kserve-7865bb985c-86pmx\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") " pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:20.642874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:20.642785    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:21.039178 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:21.039142    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"]
Oct 30 22:08:21.582153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:21.582123    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx" event={"ID":"58d9e761-6ac8-43f3-9b01-a65f033493c1","Type":"ContainerStarted","Data":"1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e"}
Oct 30 22:08:21.582651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:21.582162    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx" event={"ID":"58d9e761-6ac8-43f3-9b01-a65f033493c1","Type":"ContainerStarted","Data":"15b76550f408d7a0138d83db7b98b45b60a0e834dcdfbb456c5c2a994949afc1"}
Oct 30 22:08:21.603001 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:21.602956    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx" podStartSLOduration=1.602942522 podStartE2EDuration="1.602942522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:21.599527049 +0000 UTC m=+1720.257623269" watchObservedRunningTime="2026-10-30 22:08:21.602942522 +0000 UTC m=+1720.261038744"
Oct 30 22:08:21.643888 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:21.643862    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:30.644079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:30.644044    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:30.657068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:30.657046    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:30.677423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:30.677397    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:31.432317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:31.432269    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"]
Oct 30 22:08:31.867115 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:31.867036    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-self-signed-certs: secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:31.867528 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:31.867130    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs podName:58d9e761-6ac8-43f3-9b01-a65f033493c1 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:32.36711078 +0000 UTC m=+1731.025206995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs") pod "explicit-model-0f1f9c-kserve-7865bb985c-86pmx" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1") : secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:32.383099 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:32.383069    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-self-signed-certs: secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:32.383284 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:32.383152    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs podName:58d9e761-6ac8-43f3-9b01-a65f033493c1 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:33.383136429 +0000 UTC m=+1732.041232630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs") pod "explicit-model-0f1f9c-kserve-7865bb985c-86pmx" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1") : secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:33.399292 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:33.399263    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-self-signed-certs: secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:33.399750 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:33.399337    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs podName:58d9e761-6ac8-43f3-9b01-a65f033493c1 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:35.399322398 +0000 UTC m=+1734.057418599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs") pod "explicit-model-0f1f9c-kserve-7865bb985c-86pmx" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1") : secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:34.650587 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:34.650548    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:08:35.458923 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:35.458889    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-self-signed-certs: secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:35.459110 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:35.459008    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs podName:58d9e761-6ac8-43f3-9b01-a65f033493c1 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:39.458987986 +0000 UTC m=+1738.117084204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs") pod "explicit-model-0f1f9c-kserve-7865bb985c-86pmx" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1") : secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:39.488870 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:39.488840    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-self-signed-certs: secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:39.489270 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:39.488918    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs podName:58d9e761-6ac8-43f3-9b01-a65f033493c1 nodeName:}" failed. No retries permitted until 2026-10-30 22:08:47.488904653 +0000 UTC m=+1746.147000867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs") pod "explicit-model-0f1f9c-kserve-7865bb985c-86pmx" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1") : secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:41.782317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:41.782286    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"]
Oct 30 22:08:46.793707 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:46.793542    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv" podUID="418c8e59-6271-46db-9028-11c4c575f169" containerName="payload-processing" containerID="cri-o://1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5" gracePeriod=25
Oct 30 22:08:47.515142 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:47.515076    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-self-signed-certs: secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:47.515329 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:47.515160    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs podName:58d9e761-6ac8-43f3-9b01-a65f033493c1 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:03.515139035 +0000 UTC m=+1762.173235262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs") pod "explicit-model-0f1f9c-kserve-7865bb985c-86pmx" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1") : secret "explicit-model-0f1f9c-kserve-self-signed-certs" not found
Oct 30 22:08:47.550996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:47.550966    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:08:47.616296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:47.616261    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/418c8e59-6271-46db-9028-11c4c575f169-kube-api-access-kfbkl\" (UniqueName: \"kubernetes.io/projected/418c8e59-6271-46db-9028-11c4c575f169-kube-api-access-kfbkl\") pod \"418c8e59-6271-46db-9028-11c4c575f169\" (UID: \"418c8e59-6271-46db-9028-11c4c575f169\") "
Oct 30 22:08:47.616491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:47.616367    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/418c8e59-6271-46db-9028-11c4c575f169-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c8e59-6271-46db-9028-11c4c575f169-plugins-config-volume\") pod \"418c8e59-6271-46db-9028-11c4c575f169\" (UID: \"418c8e59-6271-46db-9028-11c4c575f169\") "
Oct 30 22:08:47.616770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:47.616750    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/418c8e59-6271-46db-9028-11c4c575f169-plugins-config-volume" pod "418c8e59-6271-46db-9028-11c4c575f169" (UID: "418c8e59-6271-46db-9028-11c4c575f169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:47.617511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:47.617441    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/418c8e59-6271-46db-9028-11c4c575f169-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:47.618817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:47.618789    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418c8e59-6271-46db-9028-11c4c575f169-kube-api-access-kfbkl" pod "418c8e59-6271-46db-9028-11c4c575f169" (UID: "418c8e59-6271-46db-9028-11c4c575f169"). InnerVolumeSpecName "kube-api-access-kfbkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:47.721635 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:47.721592    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbkl\" (UniqueName: \"kubernetes.io/projected/418c8e59-6271-46db-9028-11c4c575f169-kube-api-access-kfbkl\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:47.803458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:47.803379    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx" podUID="58d9e761-6ac8-43f3-9b01-a65f033493c1" containerName="main" containerID="cri-o://1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e" gracePeriod=45
Oct 30 22:08:48.073027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.073001    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:48.129146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.129109    4282 generic.go:356] "Generic (PLEG): container finished" podID="418c8e59-6271-46db-9028-11c4c575f169" containerID="1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5" exitCode=0
Oct 30 22:08:48.129336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.129176    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"
Oct 30 22:08:48.129336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.129226    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv" event={"ID":"418c8e59-6271-46db-9028-11c4c575f169","Type":"ContainerDied","Data":"1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5"}
Oct 30 22:08:48.129336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.129320    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv" event={"ID":"418c8e59-6271-46db-9028-11c4c575f169","Type":"ContainerDied","Data":"c2f98e7eac388fc95a93fe6cb8f68154999e86cfcaf60b17d3f2c08501d8e26c"}
Oct 30 22:08:48.129562 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.129355    4282 scope.go:122] "RemoveContainer" containerID="1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5"
Oct 30 22:08:48.133393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.133363    4282 generic.go:356] "Generic (PLEG): container finished" podID="5f841aa6-c794-4988-ac80-934f5106059d" containerID="983fdd40a3483427a7f6e20bb1065248cb64deb3b3691deaa858262f16f70315" exitCode=0
Oct 30 22:08:48.133637 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.133433    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" event={"ID":"5f841aa6-c794-4988-ac80-934f5106059d","Type":"ContainerDied","Data":"983fdd40a3483427a7f6e20bb1065248cb64deb3b3691deaa858262f16f70315"}
Oct 30 22:08:48.134375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.134352    4282 scope.go:122] "RemoveContainer" containerID="983fdd40a3483427a7f6e20bb1065248cb64deb3b3691deaa858262f16f70315"
Oct 30 22:08:48.135855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.135818    4282 generic.go:356] "Generic (PLEG): container finished" podID="58d9e761-6ac8-43f3-9b01-a65f033493c1" containerID="1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e" exitCode=0
Oct 30 22:08:48.135984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.135918    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx" event={"ID":"58d9e761-6ac8-43f3-9b01-a65f033493c1","Type":"ContainerDied","Data":"1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e"}
Oct 30 22:08:48.135984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.135952    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx" event={"ID":"58d9e761-6ac8-43f3-9b01-a65f033493c1","Type":"ContainerDied","Data":"15b76550f408d7a0138d83db7b98b45b60a0e834dcdfbb456c5c2a994949afc1"}
Oct 30 22:08:48.136121 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.135991    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"
Oct 30 22:08:48.136606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.136572    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:08:48.136730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.136665    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:08:48.136730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.136715    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:08:48.143837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.143820    4282 scope.go:122] "RemoveContainer" containerID="1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5"
Oct 30 22:08:48.144102 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:48.144080    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5\": container with ID starting with 1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5 not found: ID does not exist" containerID="1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5"
Oct 30 22:08:48.144197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.144114    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5"} err="failed to get container status \"1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5\": rpc error: code = NotFound desc = could not find container \"1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5\": container with ID starting with 1cbd825f878aceca0d94fcf9ef67f83ee31fd456cf780e8e39dd354dd9dcc0c5 not found: ID does not exist"
Oct 30 22:08:48.144197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.144135    4282 scope.go:122] "RemoveContainer" containerID="1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e"
Oct 30 22:08:48.153866 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.153838    4282 scope.go:122] "RemoveContainer" containerID="1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e"
Oct 30 22:08:48.156697 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:08:48.156592    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e\": container with ID starting with 1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e not found: ID does not exist" containerID="1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e"
Oct 30 22:08:48.156697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.156616    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e"} err="failed to get container status \"1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e\": rpc error: code = NotFound desc = could not find container \"1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e\": container with ID starting with 1ebc7c354c20621b62550d9795870c08c24a9d137869392f8359bc6546cc320e not found: ID does not exist"
Oct 30 22:08:48.179986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.179959    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"]
Oct 30 22:08:48.192099 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.192072    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0f1f9c-68b75877c5-lq6gv"]
Oct 30 22:08:48.229539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.229513    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-model-cache\") pod \"58d9e761-6ac8-43f3-9b01-a65f033493c1\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") "
Oct 30 22:08:48.229669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.229621    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-tmp-dir\") pod \"58d9e761-6ac8-43f3-9b01-a65f033493c1\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") "
Oct 30 22:08:48.229755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.229740    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-dshm\") pod \"58d9e761-6ac8-43f3-9b01-a65f033493c1\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") "
Oct 30 22:08:48.229861 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.229779    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58d9e761-6ac8-43f3-9b01-a65f033493c1-kube-api-access-l5p2n\" (UniqueName: \"kubernetes.io/projected/58d9e761-6ac8-43f3-9b01-a65f033493c1-kube-api-access-l5p2n\") pod \"58d9e761-6ac8-43f3-9b01-a65f033493c1\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") "
Oct 30 22:08:48.229905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.229868    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-model-cache" pod "58d9e761-6ac8-43f3-9b01-a65f033493c1" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:48.230004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.229984    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-home\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-home\") pod \"58d9e761-6ac8-43f3-9b01-a65f033493c1\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") "
Oct 30 22:08:48.230069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.230045    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs\") pod \"58d9e761-6ac8-43f3-9b01-a65f033493c1\" (UID: \"58d9e761-6ac8-43f3-9b01-a65f033493c1\") "
Oct 30 22:08:48.230416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.230388    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-home" pod "58d9e761-6ac8-43f3-9b01-a65f033493c1" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:48.230538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.230483    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-tmp-dir" pod "58d9e761-6ac8-43f3-9b01-a65f033493c1" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:48.231312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.231291    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:48.231428 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.231315    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:48.231428 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.231328    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:48.232791 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.232769    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d9e761-6ac8-43f3-9b01-a65f033493c1-kube-api-access-l5p2n" pod "58d9e761-6ac8-43f3-9b01-a65f033493c1" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1"). InnerVolumeSpecName "kube-api-access-l5p2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:48.233616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.233592    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-dshm" pod "58d9e761-6ac8-43f3-9b01-a65f033493c1" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:48.233701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.233660    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs" pod "58d9e761-6ac8-43f3-9b01-a65f033493c1" (UID: "58d9e761-6ac8-43f3-9b01-a65f033493c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:48.333127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.332988    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58d9e761-6ac8-43f3-9b01-a65f033493c1-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:48.333127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.333024    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58d9e761-6ac8-43f3-9b01-a65f033493c1-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:48.333127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.333039    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5p2n\" (UniqueName: \"kubernetes.io/projected/58d9e761-6ac8-43f3-9b01-a65f033493c1-kube-api-access-l5p2n\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:48.461381 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.461337    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"]
Oct 30 22:08:48.465262 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:48.465233    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0f1f9c/explicit-model-0f1f9c-kserve-7865bb985c-86pmx"]
Oct 30 22:08:49.147159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:49.147117    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" event={"ID":"5f841aa6-c794-4988-ac80-934f5106059d","Type":"ContainerStarted","Data":"45abb51f2b2f6ee8fb60360368eace8bf513012c21be6c4afdd566cf13da108d"}
Oct 30 22:08:49.484255 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:49.484218    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="418c8e59-6271-46db-9028-11c4c575f169" path="/var/lib/kubelet/pods/418c8e59-6271-46db-9028-11c4c575f169/volumes"
Oct 30 22:08:49.484811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:49.484795    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58d9e761-6ac8-43f3-9b01-a65f033493c1" path="/var/lib/kubelet/pods/58d9e761-6ac8-43f3-9b01-a65f033493c1/volumes"
Oct 30 22:08:49.886555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:49.875716    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:08:49.886555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:49.875851    4282 patch_prober.go:29] interesting pod/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:08:49.886555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:49.875897    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:08:50.875492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:50.874833    4282 patch_prober.go:29] interesting pod/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:08:50.875492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:50.874893    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:08:51.112575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:51.112541    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:08:51.875494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:51.874948    4282 patch_prober.go:29] interesting pod/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:08:51.875494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:51.875025    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:08:52.880949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:52.880905    4282 patch_prober.go:29] interesting pod/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:08:52.881654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:52.881594    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:08:53.877186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:53.877137    4282 patch_prober.go:29] interesting pod/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body=
Oct 30 22:08:53.877431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:53.877211    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused"
Oct 30 22:08:54.583304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:54.583273    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"]
Oct 30 22:08:54.583867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:54.583486    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy" containerID="cri-o://45abb51f2b2f6ee8fb60360368eace8bf513012c21be6c4afdd566cf13da108d" gracePeriod=30
Oct 30 22:08:55.759832 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.759802    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"]
Oct 30 22:08:55.760615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.760598    4282 cpu_manager.go:414] "containerMap: removing container" podUID="418c8e59-6271-46db-9028-11c4c575f169" containerName="payload-processing"
Oct 30 22:08:55.760674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.760616    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="418c8e59-6271-46db-9028-11c4c575f169" containerName="payload-processing"
Oct 30 22:08:55.760674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.760633    4282 cpu_manager.go:414] "containerMap: removing container" podUID="58d9e761-6ac8-43f3-9b01-a65f033493c1" containerName="main"
Oct 30 22:08:55.760674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.760641    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58d9e761-6ac8-43f3-9b01-a65f033493c1" containerName="main"
Oct 30 22:08:55.760770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.760750    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="418c8e59-6271-46db-9028-11c4c575f169" containerName="payload-processing"
Oct 30 22:08:55.760770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.760760    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="58d9e761-6ac8-43f3-9b01-a65f033493c1" containerName="main"
Oct 30 22:08:55.765823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.765805    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.777078 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.776977    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-e50206-openshift-default-dockercfg-p6nw8\""
Oct 30 22:08:55.779500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.779456    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"]
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905128    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-envoy\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905287    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-credential-socket\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905400    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-certs\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905437    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-podinfo\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905583    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-socket\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905676    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istiod-ca-cert\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905738    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-token\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905765    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwjh\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-kube-api-access-4jwjh\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:55.909874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:55.905799    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-data\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.008611    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-socket\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.008730    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istiod-ca-cert\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.008788    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-token\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.008813    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwjh\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-kube-api-access-4jwjh\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.008844    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-data\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.008933    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-envoy\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.009059    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-credential-socket\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.009155    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-certs\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.009762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.009188    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-podinfo\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.010617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.010544    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istiod-ca-cert\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.013502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.010821    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-socket\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.020512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.018338    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-podinfo\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.020512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.018609    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-credential-socket\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.020512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.018818    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-certs\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.020512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.019487    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-data\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.048077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.041978    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-token\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.048077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.047918    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-envoy\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.068492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.065905    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwjh\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-kube-api-access-4jwjh\") pod \"e2e-noten-e50206-openshift-default-674688bb57-xmdgh\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") " pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.084493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.083735    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:56.551159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.551134    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"]
Oct 30 22:08:56.552073 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:08:56.552046    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d07dcf_afd7_40b9_9f8e_ae382dfc7cbd.slice/crio-2d5f2c204c43b914a968fcd7135e69d765f6adb3a8666b747d9843fbdb8c02a4 WatchSource:0}: Error finding container 2d5f2c204c43b914a968fcd7135e69d765f6adb3a8666b747d9843fbdb8c02a4: Status 404 returned error can't find the container with id 2d5f2c204c43b914a968fcd7135e69d765f6adb3a8666b747d9843fbdb8c02a4
Oct 30 22:08:56.554872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.554837    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:08:56.554957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.554905    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:08:56.554957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:56.554934    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:08:57.308527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:57.305836    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh" event={"ID":"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd","Type":"ContainerStarted","Data":"2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3"}
Oct 30 22:08:57.308527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:57.305925    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh" event={"ID":"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd","Type":"ContainerStarted","Data":"2d5f2c204c43b914a968fcd7135e69d765f6adb3a8666b747d9843fbdb8c02a4"}
Oct 30 22:08:58.090500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.090254    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:58.125743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.123128    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:58.154345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.154285    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh" podStartSLOduration=3.1542674330000002 podStartE2EDuration="3.154267433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:57.356626761 +0000 UTC m=+1756.014722987" watchObservedRunningTime="2026-10-30 22:08:58.154267433 +0000 UTC m=+1756.812363658"
Oct 30 22:08:58.312986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.312956    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:58.313812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.313794    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:08:58.865313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.864143    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"]
Oct 30 22:08:58.872944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.872914    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:58.878606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.878584    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e50206\"/\"noten-model-e50206-kserve-self-signed-certs\""
Oct 30 22:08:58.880039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.880010    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"]
Oct 30 22:08:58.881383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.881359    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e50206\"/\"default-dockercfg-sbhcw\""
Oct 30 22:08:58.882795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.882553    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e50206\"/\"kube-root-ca.crt\""
Oct 30 22:08:58.882795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.882637    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e50206\"/\"openshift-service-ca.crt\""
Oct 30 22:08:58.971994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.971965    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-dshm\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:58.972178 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.972029    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:58.972178 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.972155    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-home\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:58.972291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.972215    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-tmp-dir\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:58.972403 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.972378    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnb8\" (UniqueName: \"kubernetes.io/projected/2cff85cd-57f8-4a68-91a7-7a73c6950407-kube-api-access-hxnb8\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:58.972458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:58.972429    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-model-cache\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.084491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.083617    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnb8\" (UniqueName: \"kubernetes.io/projected/2cff85cd-57f8-4a68-91a7-7a73c6950407-kube-api-access-hxnb8\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.084491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.083693    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-model-cache\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.084491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.083886    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-dshm\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.084491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.083971    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.084491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.084013    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-home\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.084491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.084073    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-tmp-dir\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.084983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.084672    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-tmp-dir\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.096490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.085246    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-model-cache\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.096490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.086088    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-home\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.104490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.101707    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.130545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.128596    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-dshm\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.130545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.129407    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnb8\" (UniqueName: \"kubernetes.io/projected/2cff85cd-57f8-4a68-91a7-7a73c6950407-kube-api-access-hxnb8\") pod \"noten-model-e50206-kserve-5f6b575957-zsp4n\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") " pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.198531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.194847    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:08:59.620375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:08:59.620349    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"]
Oct 30 22:08:59.621016 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:08:59.620995    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cff85cd_57f8_4a68_91a7_7a73c6950407.slice/crio-d9161298a640fff2e73918e2dd483d8a3b44284607744381912f213567fb8757 WatchSource:0}: Error finding container d9161298a640fff2e73918e2dd483d8a3b44284607744381912f213567fb8757: Status 404 returned error can't find the container with id d9161298a640fff2e73918e2dd483d8a3b44284607744381912f213567fb8757
Oct 30 22:09:00.444389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.435440    4282 generic.go:356] "Generic (PLEG): container finished" podID="5f841aa6-c794-4988-ac80-934f5106059d" containerID="45abb51f2b2f6ee8fb60360368eace8bf513012c21be6c4afdd566cf13da108d" exitCode=0
Oct 30 22:09:00.444389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.435601    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" event={"ID":"5f841aa6-c794-4988-ac80-934f5106059d","Type":"ContainerDied","Data":"45abb51f2b2f6ee8fb60360368eace8bf513012c21be6c4afdd566cf13da108d"}
Oct 30 22:09:00.444389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.435663    4282 scope.go:122] "RemoveContainer" containerID="983fdd40a3483427a7f6e20bb1065248cb64deb3b3691deaa858262f16f70315"
Oct 30 22:09:00.444389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.443055    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n" event={"ID":"2cff85cd-57f8-4a68-91a7-7a73c6950407","Type":"ContainerStarted","Data":"6236a9dd6f53ae45e03243c4b9cb3df42d817ccf35269041324460d13a80b753"}
Oct 30 22:09:00.444389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.443094    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n" event={"ID":"2cff85cd-57f8-4a68-91a7-7a73c6950407","Type":"ContainerStarted","Data":"d9161298a640fff2e73918e2dd483d8a3b44284607744381912f213567fb8757"}
Oct 30 22:09:00.476444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.475779    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n" podStartSLOduration=2.475764288 podStartE2EDuration="2.475764288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:00.473934369 +0000 UTC m=+1759.132030592" watchObservedRunningTime="2026-10-30 22:09:00.475764288 +0000 UTC m=+1759.133860511"
Oct 30 22:09:00.506108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.506085    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:09:00.538326 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538298    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-kube-api-access-5k9fc\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-kube-api-access-5k9fc\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.538525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538366    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-envoy\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.538525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538462    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f841aa6-c794-4988-ac80-934f5106059d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f841aa6-c794-4988-ac80-934f5106059d-istiod-ca-cert\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.538525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538506    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-istio-token\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-istio-token\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.538700 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538549    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-data\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.538700 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538649    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-certs\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.538805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538709    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-socket\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.538805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538781    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f841aa6-c794-4988-ac80-934f5106059d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f841aa6-c794-4988-ac80-934f5106059d-istio-podinfo\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.538900 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.538881    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-credential-socket\") pod \"5f841aa6-c794-4988-ac80-934f5106059d\" (UID: \"5f841aa6-c794-4988-ac80-934f5106059d\") "
Oct 30 22:09:00.539438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.539407    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-data" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:00.539747 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.539719    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-certs" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:00.540755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.540729    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:00.540849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.540761    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:00.540929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.540898    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-credential-socket" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:00.541059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.540965    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f841aa6-c794-4988-ac80-934f5106059d-istiod-ca-cert" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:00.541059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.541037    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-socket" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:00.543384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.543355    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f841aa6-c794-4988-ac80-934f5106059d-istio-podinfo" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:00.543622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.543590    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-istio-token" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:00.543622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.543591    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-envoy" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:00.544528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.544498    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-kube-api-access-5k9fc" pod "5f841aa6-c794-4988-ac80-934f5106059d" (UID: "5f841aa6-c794-4988-ac80-934f5106059d"). InnerVolumeSpecName "kube-api-access-5k9fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:00.694951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.693248    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k9fc\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-kube-api-access-5k9fc\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:00.694951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.693286    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:00.694951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.693303    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f841aa6-c794-4988-ac80-934f5106059d-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:00.694951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.693318    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f841aa6-c794-4988-ac80-934f5106059d-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:00.694951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.693336    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:00.694951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.693348    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f841aa6-c794-4988-ac80-934f5106059d-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:00.694951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:00.693361    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f841aa6-c794-4988-ac80-934f5106059d-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:01.196274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:01.196247    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:09:01.450263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:01.450173    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"
Oct 30 22:09:01.450263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:01.450212    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn" event={"ID":"5f841aa6-c794-4988-ac80-934f5106059d","Type":"ContainerDied","Data":"5a8a6bb69134e104ca55f595d4990cc9f3c319996346e7d7f8d91c8ed20b6a83"}
Oct 30 22:09:01.450530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:01.450271    4282 scope.go:122] "RemoveContainer" containerID="45abb51f2b2f6ee8fb60360368eace8bf513012c21be6c4afdd566cf13da108d"
Oct 30 22:09:01.480698 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:01.480673    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"]
Oct 30 22:09:01.485169 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:01.485150    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-0f1f9c-openshift-default-7ccd7bd845-z9hvn"]
Oct 30 22:09:03.465789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:03.465756    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f841aa6-c794-4988-ac80-934f5106059d" path="/var/lib/kubelet/pods/5f841aa6-c794-4988-ac80-934f5106059d/volumes"
Oct 30 22:09:04.650073 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:04.650036    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:09:09.050761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:09.050726    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:09:09.196069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:09.196041    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:09:09.211309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:09.211285    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:09:09.611393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:09.611364    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:09:09.852042 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:09.852010    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"]
Oct 30 22:09:09.985665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:09.982320    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"]
Oct 30 22:09:09.985665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:09.982596    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh" podUID="37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" containerName="istio-proxy" containerID="cri-o://2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3" gracePeriod=30
Oct 30 22:09:10.716448 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:10.716420    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-self-signed-certs: secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:10.717192 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:10.717168    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs podName:2cff85cd-57f8-4a68-91a7-7a73c6950407 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:11.217143442 +0000 UTC m=+1769.875239658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs") pod "noten-model-e50206-kserve-5f6b575957-zsp4n" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407") : secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:11.223721 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:11.223693    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-self-signed-certs: secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:11.223921 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:11.223762    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs podName:2cff85cd-57f8-4a68-91a7-7a73c6950407 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:12.223746761 +0000 UTC m=+1770.881842975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs") pod "noten-model-e50206-kserve-5f6b575957-zsp4n" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407") : secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:12.248608 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:12.248574    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-self-signed-certs: secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:12.248990 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:12.248643    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs podName:2cff85cd-57f8-4a68-91a7-7a73c6950407 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:14.248629946 +0000 UTC m=+1772.906726164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs") pod "noten-model-e50206-kserve-5f6b575957-zsp4n" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407") : secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:13.315692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:13.315660    4282 patch_prober.go:29] interesting pod/e2e-noten-e50206-openshift-default-674688bb57-xmdgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 30 22:09:13.316153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:13.315724    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh" podUID="37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:14.291327 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:14.291295    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-self-signed-certs: secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:14.291551 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:14.291371    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs podName:2cff85cd-57f8-4a68-91a7-7a73c6950407 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:18.291353845 +0000 UTC m=+1776.949450046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs") pod "noten-model-e50206-kserve-5f6b575957-zsp4n" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407") : secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:15.683188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.683158    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:09:15.788299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.788269    4282 generic.go:356] "Generic (PLEG): container finished" podID="37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" containerID="2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3" exitCode=0
Oct 30 22:09:15.788523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.788310    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"
Oct 30 22:09:15.788523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.788355    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh" event={"ID":"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd","Type":"ContainerDied","Data":"2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3"}
Oct 30 22:09:15.788523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.788410    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh" event={"ID":"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd","Type":"ContainerDied","Data":"2d5f2c204c43b914a968fcd7135e69d765f6adb3a8666b747d9843fbdb8c02a4"}
Oct 30 22:09:15.788523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.788431    4282 scope.go:122] "RemoveContainer" containerID="2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3"
Oct 30 22:09:15.799757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.799742    4282 scope.go:122] "RemoveContainer" containerID="2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3"
Oct 30 22:09:15.800039 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:15.800019    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3\": container with ID starting with 2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3 not found: ID does not exist" containerID="2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3"
Oct 30 22:09:15.800088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.800053    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3"} err="failed to get container status \"2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3\": rpc error: code = NotFound desc = could not find container \"2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3\": container with ID starting with 2ec1e6eccf796bc8f2b65118740b6fae3198d3ba0a696e5191e79180826eeaf3 not found: ID does not exist"
Oct 30 22:09:15.825752 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.825719    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-certs\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.825875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.825815    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-kube-api-access-4jwjh\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-kube-api-access-4jwjh\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.825949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.825885    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-credential-socket\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.826011 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.825982    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-token\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-token\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.826069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826014    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-certs" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:15.826069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826059    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-envoy\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.826180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826093    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-socket\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.826240 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826218    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-credential-socket" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:15.826375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826360    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istiod-ca-cert\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.826682 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826414    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-data\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.826682 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826415    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-socket" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:15.826682 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826455    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-podinfo\") pod \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\" (UID: \"37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd\") "
Oct 30 22:09:15.826971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.826708    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-data" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:15.827663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.827498    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:15.827663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.827526    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:15.827663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.827542    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:15.827663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.827583    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:15.827922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.827873    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istiod-ca-cert" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:15.828710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.828682    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-envoy" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:15.828835 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.828747    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-token" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:15.829268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.829248    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-podinfo" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:15.829268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.829249    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-kube-api-access-4jwjh" pod "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" (UID: "37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd"). InnerVolumeSpecName "kube-api-access-4jwjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:15.929272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.929245    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:15.929272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.929271    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:15.929444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.929280    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:15.929444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.929291    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:15.929444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:15.929300    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwjh\" (UniqueName: \"kubernetes.io/projected/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd-kube-api-access-4jwjh\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.125443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:16.125415    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"]
Oct 30 22:09:16.135438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:16.135411    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-e50206-openshift-default-674688bb57-xmdgh"]
Oct 30 22:09:17.468787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:17.468753    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" path="/var/lib/kubelet/pods/37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd/volumes"
Oct 30 22:09:18.382673 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:18.382643    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-self-signed-certs: secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:18.382866 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:18.382712    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs podName:2cff85cd-57f8-4a68-91a7-7a73c6950407 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:26.382699125 +0000 UTC m=+1785.040795326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs") pod "noten-model-e50206-kserve-5f6b575957-zsp4n" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407") : secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:26.462394 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:26.462355    4282 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-self-signed-certs: secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:26.462946 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:26.462433    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs podName:2cff85cd-57f8-4a68-91a7-7a73c6950407 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:42.462419429 +0000 UTC m=+1801.120515643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs") pod "noten-model-e50206-kserve-5f6b575957-zsp4n" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407") : secret "noten-model-e50206-kserve-self-signed-certs" not found
Oct 30 22:09:26.749318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:26.749093    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n" podUID="2cff85cd-57f8-4a68-91a7-7a73c6950407" containerName="main" containerID="cri-o://6236a9dd6f53ae45e03243c4b9cb3df42d817ccf35269041324460d13a80b753" gracePeriod=45
Oct 30 22:09:26.908551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:26.908505    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n" event={"ID":"2cff85cd-57f8-4a68-91a7-7a73c6950407","Type":"ContainerDied","Data":"6236a9dd6f53ae45e03243c4b9cb3df42d817ccf35269041324460d13a80b753"}
Oct 30 22:09:26.908551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:26.908523    4282 generic.go:356] "Generic (PLEG): container finished" podID="2cff85cd-57f8-4a68-91a7-7a73c6950407" containerID="6236a9dd6f53ae45e03243c4b9cb3df42d817ccf35269041324460d13a80b753" exitCode=0
Oct 30 22:09:27.046986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.046960    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:09:27.082245 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.081902    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cff85cd-57f8-4a68-91a7-7a73c6950407-kube-api-access-hxnb8\" (UniqueName: \"kubernetes.io/projected/2cff85cd-57f8-4a68-91a7-7a73c6950407-kube-api-access-hxnb8\") pod \"2cff85cd-57f8-4a68-91a7-7a73c6950407\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") "
Oct 30 22:09:27.082245 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.082118    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs\") pod \"2cff85cd-57f8-4a68-91a7-7a73c6950407\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") "
Oct 30 22:09:27.082551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.082404    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-tmp-dir\") pod \"2cff85cd-57f8-4a68-91a7-7a73c6950407\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") "
Oct 30 22:09:27.082656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.082631    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-home\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-home\") pod \"2cff85cd-57f8-4a68-91a7-7a73c6950407\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") "
Oct 30 22:09:27.082819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.082724    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-dshm\") pod \"2cff85cd-57f8-4a68-91a7-7a73c6950407\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") "
Oct 30 22:09:27.082819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.082778    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-model-cache\") pod \"2cff85cd-57f8-4a68-91a7-7a73c6950407\" (UID: \"2cff85cd-57f8-4a68-91a7-7a73c6950407\") "
Oct 30 22:09:27.083285 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.083259    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-tmp-dir" pod "2cff85cd-57f8-4a68-91a7-7a73c6950407" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:27.083711 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.083685    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-model-cache" pod "2cff85cd-57f8-4a68-91a7-7a73c6950407" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:27.083711 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.083379    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-home" pod "2cff85cd-57f8-4a68-91a7-7a73c6950407" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:27.083994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.083974    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:27.085999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.085964    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cff85cd-57f8-4a68-91a7-7a73c6950407-kube-api-access-hxnb8" pod "2cff85cd-57f8-4a68-91a7-7a73c6950407" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407"). InnerVolumeSpecName "kube-api-access-hxnb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:27.091052 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.091026    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-dshm" pod "2cff85cd-57f8-4a68-91a7-7a73c6950407" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:27.103483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.092400    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs" pod "2cff85cd-57f8-4a68-91a7-7a73c6950407" (UID: "2cff85cd-57f8-4a68-91a7-7a73c6950407"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:27.185625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.185595    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:27.185625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.185620    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:27.185819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.185630    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxnb8\" (UniqueName: \"kubernetes.io/projected/2cff85cd-57f8-4a68-91a7-7a73c6950407-kube-api-access-hxnb8\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:27.185819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.185644    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cff85cd-57f8-4a68-91a7-7a73c6950407-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:27.185819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.185659    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cff85cd-57f8-4a68-91a7-7a73c6950407-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:27.918061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.918032    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"
Oct 30 22:09:27.918583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.918065    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n" event={"ID":"2cff85cd-57f8-4a68-91a7-7a73c6950407","Type":"ContainerDied","Data":"d9161298a640fff2e73918e2dd483d8a3b44284607744381912f213567fb8757"}
Oct 30 22:09:27.918583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.918116    4282 scope.go:122] "RemoveContainer" containerID="6236a9dd6f53ae45e03243c4b9cb3df42d817ccf35269041324460d13a80b753"
Oct 30 22:09:27.936369 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.936340    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"]
Oct 30 22:09:27.940307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:27.940282    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-e50206/noten-model-e50206-kserve-5f6b575957-zsp4n"]
Oct 30 22:09:29.465487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:29.465443    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cff85cd-57f8-4a68-91a7-7a73c6950407" path="/var/lib/kubelet/pods/2cff85cd-57f8-4a68-91a7-7a73c6950407/volumes"
Oct 30 22:09:34.650755 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:34.650723    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:09:37.120392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.120349    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-7a1873"]
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.140637    4282 cpu_manager.go:414] "containerMap: removing container" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150230    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150264    4282 cpu_manager.go:414] "containerMap: removing container" podUID="37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" containerName="istio-proxy"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150270    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" containerName="istio-proxy"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150326    4282 cpu_manager.go:414] "containerMap: removing container" podUID="2cff85cd-57f8-4a68-91a7-7a73c6950407" containerName="main"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150332    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cff85cd-57f8-4a68-91a7-7a73c6950407" containerName="main"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150555    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150567    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d07dcf-afd7-40b9-9f8e-ae382dfc7cbd" containerName="istio-proxy"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150584    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cff85cd-57f8-4a68-91a7-7a73c6950407" containerName="main"
Oct 30 22:09:37.152235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.150593    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy"
Oct 30 22:09:37.154507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.154449    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-7a1873"
Oct 30 22:09:37.160654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.160630    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-7a1873"]
Oct 30 22:09:37.160910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.160892    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sfm7d\""
Oct 30 22:09:37.206824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.206793    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctmv\" (UniqueName: \"kubernetes.io/projected/adb9e529-f451-458c-92b3-02a36bd92c29-kube-api-access-9ctmv\") pod \"test-curl-7697-7a1873\" (UID: \"adb9e529-f451-458c-92b3-02a36bd92c29\") " pod="opendatahub/test-curl-7697-7a1873"
Oct 30 22:09:37.326496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.311025    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctmv\" (UniqueName: \"kubernetes.io/projected/adb9e529-f451-458c-92b3-02a36bd92c29-kube-api-access-9ctmv\") pod \"test-curl-7697-7a1873\" (UID: \"adb9e529-f451-458c-92b3-02a36bd92c29\") " pod="opendatahub/test-curl-7697-7a1873"
Oct 30 22:09:37.367669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.361106    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctmv\" (UniqueName: \"kubernetes.io/projected/adb9e529-f451-458c-92b3-02a36bd92c29-kube-api-access-9ctmv\") pod \"test-curl-7697-7a1873\" (UID: \"adb9e529-f451-458c-92b3-02a36bd92c29\") " pod="opendatahub/test-curl-7697-7a1873"
Oct 30 22:09:37.473630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.473598    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-7a1873"
Oct 30 22:09:37.759904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:37.759878    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-7a1873"]
Oct 30 22:09:37.760757 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:09:37.760724    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadb9e529_f451_458c_92b3_02a36bd92c29.slice/crio-7b6c141db082711282020250e4828e6b629482a06b8f4e2020ffb9196ec14462 WatchSource:0}: Error finding container 7b6c141db082711282020250e4828e6b629482a06b8f4e2020ffb9196ec14462: Status 404 returned error can't find the container with id 7b6c141db082711282020250e4828e6b629482a06b8f4e2020ffb9196ec14462
Oct 30 22:09:38.031108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:38.030290    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-7a1873" event={"ID":"adb9e529-f451-458c-92b3-02a36bd92c29","Type":"ContainerStarted","Data":"bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c"}
Oct 30 22:09:38.031108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:38.030345    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-7a1873" event={"ID":"adb9e529-f451-458c-92b3-02a36bd92c29","Type":"ContainerStarted","Data":"7b6c141db082711282020250e4828e6b629482a06b8f4e2020ffb9196ec14462"}
Oct 30 22:09:38.049113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:38.049054    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7697-7a1873" podStartSLOduration=1.049040136 podStartE2EDuration="1.049040136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:38.045084224 +0000 UTC m=+1796.703180446" watchObservedRunningTime="2026-10-30 22:09:38.049040136 +0000 UTC m=+1796.707136353"
Oct 30 22:09:38.929491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:38.926634    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-7a1873"]
Oct 30 22:09:39.281932 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.281828    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-7a1873"]
Oct 30 22:09:39.287631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.287603    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-562940"]
Oct 30 22:09:39.288646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.288620    4282 cpu_manager.go:414] "containerMap: removing container" podUID="adb9e529-f451-458c-92b3-02a36bd92c29" containerName="test-curl-7697-7a1873"
Oct 30 22:09:39.288646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.288644    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb9e529-f451-458c-92b3-02a36bd92c29" containerName="test-curl-7697-7a1873"
Oct 30 22:09:39.288841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.288671    4282 cpu_manager.go:414] "containerMap: removing container" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy"
Oct 30 22:09:39.288841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.288679    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f841aa6-c794-4988-ac80-934f5106059d" containerName="istio-proxy"
Oct 30 22:09:39.288982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.288882    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb9e529-f451-458c-92b3-02a36bd92c29" containerName="test-curl-7697-7a1873"
Oct 30 22:09:39.292865 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.292843    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-562940"
Oct 30 22:09:39.294597 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.294571    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-562940"]
Oct 30 22:09:39.500900 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.500833    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jvk\" (UniqueName: \"kubernetes.io/projected/61b2878d-a817-4262-a06f-d58329b75e71-kube-api-access-t4jvk\") pod \"test-curl-7697-562940\" (UID: \"61b2878d-a817-4262-a06f-d58329b75e71\") " pod="opendatahub/test-curl-7697-562940"
Oct 30 22:09:39.605590 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.603003    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jvk\" (UniqueName: \"kubernetes.io/projected/61b2878d-a817-4262-a06f-d58329b75e71-kube-api-access-t4jvk\") pod \"test-curl-7697-562940\" (UID: \"61b2878d-a817-4262-a06f-d58329b75e71\") " pod="opendatahub/test-curl-7697-562940"
Oct 30 22:09:39.619550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.619305    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jvk\" (UniqueName: \"kubernetes.io/projected/61b2878d-a817-4262-a06f-d58329b75e71-kube-api-access-t4jvk\") pod \"test-curl-7697-562940\" (UID: \"61b2878d-a817-4262-a06f-d58329b75e71\") " pod="opendatahub/test-curl-7697-562940"
Oct 30 22:09:39.909800 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:39.909720    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-562940"
Oct 30 22:09:40.194505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:40.194452    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-562940"]
Oct 30 22:09:40.195068 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:09:40.195047    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61b2878d_a817_4262_a06f_d58329b75e71.slice/crio-0328aa5e6e7c863e791898075c82d2c5bd506072a49e065086c3ee0f17ecdb41 WatchSource:0}: Error finding container 0328aa5e6e7c863e791898075c82d2c5bd506072a49e065086c3ee0f17ecdb41: Status 404 returned error can't find the container with id 0328aa5e6e7c863e791898075c82d2c5bd506072a49e065086c3ee0f17ecdb41
Oct 30 22:09:40.286918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:40.286886    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-562940" event={"ID":"61b2878d-a817-4262-a06f-d58329b75e71","Type":"ContainerStarted","Data":"341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0"}
Oct 30 22:09:40.287041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:40.286945    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-562940" event={"ID":"61b2878d-a817-4262-a06f-d58329b75e71","Type":"ContainerStarted","Data":"0328aa5e6e7c863e791898075c82d2c5bd506072a49e065086c3ee0f17ecdb41"}
Oct 30 22:09:40.287041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:40.286967    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7697-7a1873" podUID="adb9e529-f451-458c-92b3-02a36bd92c29" containerName="test-curl-7697-7a1873" containerID="cri-o://bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c" gracePeriod=2
Oct 30 22:09:40.303632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:40.303584    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7697-562940" podStartSLOduration=1.3035712689999999 podStartE2EDuration="1.303571269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:40.300701254 +0000 UTC m=+1798.958797476" watchObservedRunningTime="2026-10-30 22:09:40.303571269 +0000 UTC m=+1798.961667492"
Oct 30 22:09:41.124737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.124673    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-562940"]
Oct 30 22:09:41.130371 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.130336    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-562940"]
Oct 30 22:09:41.280146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.280117    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-16c2eb"]
Oct 30 22:09:41.280962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.280942    4282 cpu_manager.go:414] "containerMap: removing container" podUID="61b2878d-a817-4262-a06f-d58329b75e71" containerName="test-curl-7697-562940"
Oct 30 22:09:41.281018 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.280965    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b2878d-a817-4262-a06f-d58329b75e71" containerName="test-curl-7697-562940"
Oct 30 22:09:41.281086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.281077    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b2878d-a817-4262-a06f-d58329b75e71" containerName="test-curl-7697-562940"
Oct 30 22:09:41.284698 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.284677    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:41.304981 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.293066    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-16c2eb"]
Oct 30 22:09:41.304981 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.303705    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"]
Oct 30 22:09:41.304981 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.303906    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy" containerID="cri-o://4fe9b9c4f9e0604fdd13e6f519af445ec1a76ec025f46f076c406e9b6fbad25b" gracePeriod=30
Oct 30 22:09:41.350331 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.350295    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjt29\" (UniqueName: \"kubernetes.io/projected/45fe35df-9ab8-4604-9057-137e82b20be4-kube-api-access-wjt29\") pod \"test-curl-7697-16c2eb\" (UID: \"45fe35df-9ab8-4604-9057-137e82b20be4\") " pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:41.451860 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.451828    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjt29\" (UniqueName: \"kubernetes.io/projected/45fe35df-9ab8-4604-9057-137e82b20be4-kube-api-access-wjt29\") pod \"test-curl-7697-16c2eb\" (UID: \"45fe35df-9ab8-4604-9057-137e82b20be4\") " pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:41.464122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.464087    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjt29\" (UniqueName: \"kubernetes.io/projected/45fe35df-9ab8-4604-9057-137e82b20be4-kube-api-access-wjt29\") pod \"test-curl-7697-16c2eb\" (UID: \"45fe35df-9ab8-4604-9057-137e82b20be4\") " pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:41.613168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.613101    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:41.969056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:41.969028    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-16c2eb"]
Oct 30 22:09:42.396881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:42.396834    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7697-562940" podUID="61b2878d-a817-4262-a06f-d58329b75e71" containerName="test-curl-7697-562940" containerID="cri-o://341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0" gracePeriod=2
Oct 30 22:09:42.397684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:42.397658    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-16c2eb" event={"ID":"45fe35df-9ab8-4604-9057-137e82b20be4","Type":"ContainerStarted","Data":"714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd"}
Oct 30 22:09:42.397741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:42.397709    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-16c2eb" event={"ID":"45fe35df-9ab8-4604-9057-137e82b20be4","Type":"ContainerStarted","Data":"c314e20badf187c81c4572c78de51e610cbb0cd78eff8e2b5484fbd5910d1bd8"}
Oct 30 22:09:42.414386 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:42.414341    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7697-16c2eb" podStartSLOduration=1.4143275229999999 podStartE2EDuration="1.414327523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:42.412849552 +0000 UTC m=+1801.070945772" watchObservedRunningTime="2026-10-30 22:09:42.414327523 +0000 UTC m=+1801.072423746"
Oct 30 22:09:42.566236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:42.566212    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-7a1873"
Oct 30 22:09:42.597076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:42.597050    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb9e529-f451-458c-92b3-02a36bd92c29-kube-api-access-9ctmv\" (UniqueName: \"kubernetes.io/projected/adb9e529-f451-458c-92b3-02a36bd92c29-kube-api-access-9ctmv\") pod \"adb9e529-f451-458c-92b3-02a36bd92c29\" (UID: \"adb9e529-f451-458c-92b3-02a36bd92c29\") "
Oct 30 22:09:42.599273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:42.599245    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb9e529-f451-458c-92b3-02a36bd92c29-kube-api-access-9ctmv" pod "adb9e529-f451-458c-92b3-02a36bd92c29" (UID: "adb9e529-f451-458c-92b3-02a36bd92c29"). InnerVolumeSpecName "kube-api-access-9ctmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:42.698981 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:42.698944    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ctmv\" (UniqueName: \"kubernetes.io/projected/adb9e529-f451-458c-92b3-02a36bd92c29-kube-api-access-9ctmv\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:43.157598 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.157533    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-16c2eb"]
Oct 30 22:09:43.194215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.191383    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-16c2eb"]
Oct 30 22:09:43.275903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.275872    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-2f1e5f"]
Oct 30 22:09:43.276663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.276646    4282 cpu_manager.go:414] "containerMap: removing container" podUID="45fe35df-9ab8-4604-9057-137e82b20be4" containerName="test-curl-7697-16c2eb"
Oct 30 22:09:43.276719 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.276664    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fe35df-9ab8-4604-9057-137e82b20be4" containerName="test-curl-7697-16c2eb"
Oct 30 22:09:43.276771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.276762    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="45fe35df-9ab8-4604-9057-137e82b20be4" containerName="test-curl-7697-16c2eb"
Oct 30 22:09:43.280535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.280512    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-2f1e5f"
Oct 30 22:09:43.289403 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.289129    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-2f1e5f"]
Oct 30 22:09:43.320541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.320515    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj6sk\" (UniqueName: \"kubernetes.io/projected/c1c5b0d0-4a6e-4761-aa61-fddca0ec702f-kube-api-access-bj6sk\") pod \"test-curl-7697-2f1e5f\" (UID: \"c1c5b0d0-4a6e-4761-aa61-fddca0ec702f\") " pod="opendatahub/test-curl-7697-2f1e5f"
Oct 30 22:09:43.406010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.405976    4282 generic.go:356] "Generic (PLEG): container finished" podID="adb9e529-f451-458c-92b3-02a36bd92c29" containerID="bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c" exitCode=137
Oct 30 22:09:43.406506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.406031    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-7a1873"
Oct 30 22:09:43.406506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.406054    4282 scope.go:122] "RemoveContainer" containerID="bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c"
Oct 30 22:09:43.417566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.417548    4282 scope.go:122] "RemoveContainer" containerID="bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c"
Oct 30 22:09:43.417818 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:43.417793    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7697-7a1873\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="adb9e529-f451-458c-92b3-02a36bd92c29" pod="opendatahub/test-curl-7697-7a1873"
Oct 30 22:09:43.417886 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:43.417830    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c\": container with ID starting with bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c not found: ID does not exist" containerID="bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c"
Oct 30 22:09:43.417886 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.417854    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c"} err="failed to get container status \"bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c\": rpc error: code = NotFound desc = could not find container \"bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c\": container with ID starting with bd9aed062adde282ac3b31975b663e626b98d34d58b3e5bb36385dad49cb896c not found: ID does not exist"
Oct 30 22:09:43.421693 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.421674    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj6sk\" (UniqueName: \"kubernetes.io/projected/c1c5b0d0-4a6e-4761-aa61-fddca0ec702f-kube-api-access-bj6sk\") pod \"test-curl-7697-2f1e5f\" (UID: \"c1c5b0d0-4a6e-4761-aa61-fddca0ec702f\") " pod="opendatahub/test-curl-7697-2f1e5f"
Oct 30 22:09:43.432842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.432821    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj6sk\" (UniqueName: \"kubernetes.io/projected/c1c5b0d0-4a6e-4761-aa61-fddca0ec702f-kube-api-access-bj6sk\") pod \"test-curl-7697-2f1e5f\" (UID: \"c1c5b0d0-4a6e-4761-aa61-fddca0ec702f\") " pod="opendatahub/test-curl-7697-2f1e5f"
Oct 30 22:09:43.464703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.464677    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb9e529-f451-458c-92b3-02a36bd92c29" path="/var/lib/kubelet/pods/adb9e529-f451-458c-92b3-02a36bd92c29/volumes"
Oct 30 22:09:43.603734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.603712    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-2f1e5f"
Oct 30 22:09:43.934143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:43.934116    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-2f1e5f"]
Oct 30 22:09:43.935215 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:09:43.935191    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1c5b0d0_4a6e_4761_aa61_fddca0ec702f.slice/crio-ee83a1442e7fba05f942abff3412d32893868726095e35c762a1e01b3726f14c WatchSource:0}: Error finding container ee83a1442e7fba05f942abff3412d32893868726095e35c762a1e01b3726f14c: Status 404 returned error can't find the container with id ee83a1442e7fba05f942abff3412d32893868726095e35c762a1e01b3726f14c
Oct 30 22:09:44.415931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:44.415893    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-2f1e5f" event={"ID":"c1c5b0d0-4a6e-4761-aa61-fddca0ec702f","Type":"ContainerStarted","Data":"8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812"}
Oct 30 22:09:44.416327 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:44.415942    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-2f1e5f" event={"ID":"c1c5b0d0-4a6e-4761-aa61-fddca0ec702f","Type":"ContainerStarted","Data":"ee83a1442e7fba05f942abff3412d32893868726095e35c762a1e01b3726f14c"}
Oct 30 22:09:44.416327 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:44.416001    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7697-16c2eb" podUID="45fe35df-9ab8-4604-9057-137e82b20be4" containerName="test-curl-7697-16c2eb" containerID="cri-o://714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd" gracePeriod=2
Oct 30 22:09:44.433722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:44.433677    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7697-2f1e5f" podStartSLOduration=1.433664121 podStartE2EDuration="1.433664121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:44.429598395 +0000 UTC m=+1803.087694618" watchObservedRunningTime="2026-10-30 22:09:44.433664121 +0000 UTC m=+1803.091760344"
Oct 30 22:09:44.614314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:44.614287    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-562940"
Oct 30 22:09:44.655485 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:44.655393    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b2878d-a817-4262-a06f-d58329b75e71-kube-api-access-t4jvk\" (UniqueName: \"kubernetes.io/projected/61b2878d-a817-4262-a06f-d58329b75e71-kube-api-access-t4jvk\") pod \"61b2878d-a817-4262-a06f-d58329b75e71\" (UID: \"61b2878d-a817-4262-a06f-d58329b75e71\") "
Oct 30 22:09:44.657891 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:44.657864    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b2878d-a817-4262-a06f-d58329b75e71-kube-api-access-t4jvk" pod "61b2878d-a817-4262-a06f-d58329b75e71" (UID: "61b2878d-a817-4262-a06f-d58329b75e71"). InnerVolumeSpecName "kube-api-access-t4jvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:44.758493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:44.758278    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jvk\" (UniqueName: \"kubernetes.io/projected/61b2878d-a817-4262-a06f-d58329b75e71-kube-api-access-t4jvk\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:45.195652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.195619    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-2f1e5f"]
Oct 30 22:09:45.199516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.199488    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-2f1e5f"]
Oct 30 22:09:45.352980 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.352945    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7697-cacf24"]
Oct 30 22:09:45.354029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.353998    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c1c5b0d0-4a6e-4761-aa61-fddca0ec702f" containerName="test-curl-7697-2f1e5f"
Oct 30 22:09:45.354029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.354028    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c5b0d0-4a6e-4761-aa61-fddca0ec702f" containerName="test-curl-7697-2f1e5f"
Oct 30 22:09:45.360523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.354263    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c5b0d0-4a6e-4761-aa61-fddca0ec702f" containerName="test-curl-7697-2f1e5f"
Oct 30 22:09:45.365601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.365579    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-cacf24"]
Oct 30 22:09:45.365761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.365696    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-cacf24"
Oct 30 22:09:45.427509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.427442    4282 generic.go:356] "Generic (PLEG): container finished" podID="61b2878d-a817-4262-a06f-d58329b75e71" containerID="341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0" exitCode=137
Oct 30 22:09:45.427964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.427523    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-562940"
Oct 30 22:09:45.427964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.427532    4282 scope.go:122] "RemoveContainer" containerID="341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0"
Oct 30 22:09:45.438667 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.438645    4282 scope.go:122] "RemoveContainer" containerID="341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0"
Oct 30 22:09:45.439567 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:45.439537    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0\": container with ID starting with 341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0 not found: ID does not exist" containerID="341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0"
Oct 30 22:09:45.439685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.439576    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0"} err="failed to get container status \"341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0\": rpc error: code = NotFound desc = could not find container \"341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0\": container with ID starting with 341eb35873344667fee21d1d45a3b820254c57f31e615bec1930b1bf879e6ea0 not found: ID does not exist"
Oct 30 22:09:45.439685 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:45.439603    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7697-562940\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="61b2878d-a817-4262-a06f-d58329b75e71" pod="opendatahub/test-curl-7697-562940"
Oct 30 22:09:45.467548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.467457    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b2878d-a817-4262-a06f-d58329b75e71" path="/var/lib/kubelet/pods/61b2878d-a817-4262-a06f-d58329b75e71/volumes"
Oct 30 22:09:45.469171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.469146    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rktq8\" (UniqueName: \"kubernetes.io/projected/90712683-884e-45fb-a88b-79f48eb6a49a-kube-api-access-rktq8\") pod \"test-curl-7697-cacf24\" (UID: \"90712683-884e-45fb-a88b-79f48eb6a49a\") " pod="opendatahub/test-curl-7697-cacf24"
Oct 30 22:09:45.575875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.571968    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rktq8\" (UniqueName: \"kubernetes.io/projected/90712683-884e-45fb-a88b-79f48eb6a49a-kube-api-access-rktq8\") pod \"test-curl-7697-cacf24\" (UID: \"90712683-884e-45fb-a88b-79f48eb6a49a\") " pod="opendatahub/test-curl-7697-cacf24"
Oct 30 22:09:45.603215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.599527    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rktq8\" (UniqueName: \"kubernetes.io/projected/90712683-884e-45fb-a88b-79f48eb6a49a-kube-api-access-rktq8\") pod \"test-curl-7697-cacf24\" (UID: \"90712683-884e-45fb-a88b-79f48eb6a49a\") " pod="opendatahub/test-curl-7697-cacf24"
Oct 30 22:09:45.685454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.685425    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-cacf24"
Oct 30 22:09:45.969800 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:45.969771    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7697-cacf24"]
Oct 30 22:09:45.971452 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:09:45.971421    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90712683_884e_45fb_a88b_79f48eb6a49a.slice/crio-528137663ad06bb5789478ce506df238fd0d5f4de35f2a55abc10982bd6d2adc WatchSource:0}: Error finding container 528137663ad06bb5789478ce506df238fd0d5f4de35f2a55abc10982bd6d2adc: Status 404 returned error can't find the container with id 528137663ad06bb5789478ce506df238fd0d5f4de35f2a55abc10982bd6d2adc
Oct 30 22:09:46.438240 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.438205    4282 generic.go:356] "Generic (PLEG): container finished" podID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerID="4fe9b9c4f9e0604fdd13e6f519af445ec1a76ec025f46f076c406e9b6fbad25b" exitCode=0
Oct 30 22:09:46.438715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.438298    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" event={"ID":"d159aeca-66aa-4b6b-b7cb-720375678e70","Type":"ContainerDied","Data":"4fe9b9c4f9e0604fdd13e6f519af445ec1a76ec025f46f076c406e9b6fbad25b"}
Oct 30 22:09:46.440168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.440141    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-cacf24" event={"ID":"90712683-884e-45fb-a88b-79f48eb6a49a","Type":"ContainerStarted","Data":"6c5510cfbdc9cfb48ca2b05552dd8e00d79bc2ddfc579fa92b3340ab3f413787"}
Oct 30 22:09:46.440276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.440182    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7697-cacf24" event={"ID":"90712683-884e-45fb-a88b-79f48eb6a49a","Type":"ContainerStarted","Data":"528137663ad06bb5789478ce506df238fd0d5f4de35f2a55abc10982bd6d2adc"}
Oct 30 22:09:46.440276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.440193    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7697-2f1e5f" podUID="c1c5b0d0-4a6e-4761-aa61-fddca0ec702f" containerName="test-curl-7697-2f1e5f" containerID="cri-o://8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812" gracePeriod=2
Oct 30 22:09:46.624588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.624561    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:46.707567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.707533    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45fe35df-9ab8-4604-9057-137e82b20be4-kube-api-access-wjt29\" (UniqueName: \"kubernetes.io/projected/45fe35df-9ab8-4604-9057-137e82b20be4-kube-api-access-wjt29\") pod \"45fe35df-9ab8-4604-9057-137e82b20be4\" (UID: \"45fe35df-9ab8-4604-9057-137e82b20be4\") "
Oct 30 22:09:46.710293 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.710255    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fe35df-9ab8-4604-9057-137e82b20be4-kube-api-access-wjt29" pod "45fe35df-9ab8-4604-9057-137e82b20be4" (UID: "45fe35df-9ab8-4604-9057-137e82b20be4"). InnerVolumeSpecName "kube-api-access-wjt29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:46.826493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:46.814195    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjt29\" (UniqueName: \"kubernetes.io/projected/45fe35df-9ab8-4604-9057-137e82b20be4-kube-api-access-wjt29\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.065382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.065358    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:09:47.100837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.100767    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7697-cacf24" podStartSLOduration=2.100748596 podStartE2EDuration="2.100748596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:46.457000663 +0000 UTC m=+1805.115096882" watchObservedRunningTime="2026-10-30 22:09:47.100748596 +0000 UTC m=+1805.758844822"
Oct 30 22:09:47.118938 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.118907    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d159aeca-66aa-4b6b-b7cb-720375678e70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159aeca-66aa-4b6b-b7cb-720375678e70-istiod-ca-cert\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.118969    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-kube-api-access-7b65g\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-kube-api-access-7b65g\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119159    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-certs\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119212    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-data\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119239    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-envoy\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119277    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-podinfo\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119396    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-credential-socket\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119448    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-socket\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119524    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-token\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-token\") pod \"d159aeca-66aa-4b6b-b7cb-720375678e70\" (UID: \"d159aeca-66aa-4b6b-b7cb-720375678e70\") "
Oct 30 22:09:47.119947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119915    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d159aeca-66aa-4b6b-b7cb-720375678e70-istiod-ca-cert" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:47.119947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.119935    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-certs" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:47.120200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.120177    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-credential-socket" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:47.120515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.120446    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.120515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.120491    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d159aeca-66aa-4b6b-b7cb-720375678e70-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.120515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.120512    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.120709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.120598    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-socket" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:47.120943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.120912    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-data" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:47.123783 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.123287    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-podinfo" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:47.123783 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.123338    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-kube-api-access-7b65g" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "kube-api-access-7b65g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:47.123783 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.123371    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-token" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:47.124514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.124439    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-envoy" pod "d159aeca-66aa-4b6b-b7cb-720375678e70" (UID: "d159aeca-66aa-4b6b-b7cb-720375678e70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:47.197267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.197182    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7697-cacf24"]
Oct 30 22:09:47.221878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.221842    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.221878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.221873    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.221878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.221884    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.222115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.221893    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d159aeca-66aa-4b6b-b7cb-720375678e70-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.222115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.221902    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.222115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.221911    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b65g\" (UniqueName: \"kubernetes.io/projected/d159aeca-66aa-4b6b-b7cb-720375678e70-kube-api-access-7b65g\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:47.222628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.222596    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7697-cacf24"]
Oct 30 22:09:47.448785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.448698    4282 generic.go:356] "Generic (PLEG): container finished" podID="45fe35df-9ab8-4604-9057-137e82b20be4" containerID="714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd" exitCode=137
Oct 30 22:09:47.448785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.448754    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:47.448785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.448783    4282 scope.go:122] "RemoveContainer" containerID="714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd"
Oct 30 22:09:47.450653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.450620    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"
Oct 30 22:09:47.450653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.450640    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d" event={"ID":"d159aeca-66aa-4b6b-b7cb-720375678e70","Type":"ContainerDied","Data":"ae9a55517dc2234870c5fa119c4fea2013a8b4b1f7d1be3cc793a1091b3d3eb0"}
Oct 30 22:09:47.459379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.459359    4282 scope.go:122] "RemoveContainer" containerID="714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd"
Oct 30 22:09:47.459772 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:47.459748    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd\": container with ID starting with 714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd not found: ID does not exist" containerID="714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd"
Oct 30 22:09:47.459884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.459779    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd"} err="failed to get container status \"714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd\": rpc error: code = NotFound desc = could not find container \"714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd\": container with ID starting with 714a31fb921e8bfba952760cdb2d7a793e15151839ff586085536170246f4abd not found: ID does not exist"
Oct 30 22:09:47.459884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.459798    4282 scope.go:122] "RemoveContainer" containerID="4fe9b9c4f9e0604fdd13e6f519af445ec1a76ec025f46f076c406e9b6fbad25b"
Oct 30 22:09:47.460659 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:47.460630    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7697-16c2eb\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="45fe35df-9ab8-4604-9057-137e82b20be4" pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:47.466086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.466063    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45fe35df-9ab8-4604-9057-137e82b20be4" path="/var/lib/kubelet/pods/45fe35df-9ab8-4604-9057-137e82b20be4/volumes"
Oct 30 22:09:47.503708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.502574    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"]
Oct 30 22:09:47.504215 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:47.504179    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7697-16c2eb\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="45fe35df-9ab8-4604-9057-137e82b20be4" pod="opendatahub/test-curl-7697-16c2eb"
Oct 30 22:09:47.515077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:47.515008    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-s4t2d"]
Oct 30 22:09:48.459564 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:48.459535    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7697-cacf24" podUID="90712683-884e-45fb-a88b-79f48eb6a49a" containerName="test-curl-7697-cacf24" containerID="cri-o://6c5510cfbdc9cfb48ca2b05552dd8e00d79bc2ddfc579fa92b3340ab3f413787" gracePeriod=2
Oct 30 22:09:48.653568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:48.653543    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-2f1e5f"
Oct 30 22:09:48.753279 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:48.753200    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c5b0d0-4a6e-4761-aa61-fddca0ec702f-kube-api-access-bj6sk\" (UniqueName: \"kubernetes.io/projected/c1c5b0d0-4a6e-4761-aa61-fddca0ec702f-kube-api-access-bj6sk\") pod \"c1c5b0d0-4a6e-4761-aa61-fddca0ec702f\" (UID: \"c1c5b0d0-4a6e-4761-aa61-fddca0ec702f\") "
Oct 30 22:09:48.755481 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:48.755441    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c5b0d0-4a6e-4761-aa61-fddca0ec702f-kube-api-access-bj6sk" pod "c1c5b0d0-4a6e-4761-aa61-fddca0ec702f" (UID: "c1c5b0d0-4a6e-4761-aa61-fddca0ec702f"). InnerVolumeSpecName "kube-api-access-bj6sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:48.862489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:48.857793    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj6sk\" (UniqueName: \"kubernetes.io/projected/c1c5b0d0-4a6e-4761-aa61-fddca0ec702f-kube-api-access-bj6sk\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:49.297020 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.296979    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"]
Oct 30 22:09:49.298058 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.298035    4282 cpu_manager.go:414] "containerMap: removing container" podUID="90712683-884e-45fb-a88b-79f48eb6a49a" containerName="test-curl-7697-cacf24"
Oct 30 22:09:49.298131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.298072    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90712683-884e-45fb-a88b-79f48eb6a49a" containerName="test-curl-7697-cacf24"
Oct 30 22:09:49.298172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.298147    4282 cpu_manager.go:414] "containerMap: removing container" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy"
Oct 30 22:09:49.298172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.298157    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy"
Oct 30 22:09:49.298351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.298338    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="90712683-884e-45fb-a88b-79f48eb6a49a" containerName="test-curl-7697-cacf24"
Oct 30 22:09:49.298386 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.298364    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" containerName="istio-proxy"
Oct 30 22:09:49.302167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.302153    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.310048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.310007    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a39158-openshift-default-dockercfg-xz2kf\""
Oct 30 22:09:49.312955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.312926    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"]
Oct 30 22:09:49.371419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371386    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmqxv\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-kube-api-access-dmqxv\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.371628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371446    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/579717b1-feef-4e2a-9da8-62721859adeb-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.371628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371583    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/579717b1-feef-4e2a-9da8-62721859adeb-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.371766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371637    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.371766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371655    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.371766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371726    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.371766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371760    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.372000 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371914    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.372000 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.371939    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.493656    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.493708    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.493753    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.493796    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.493917    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.493966    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.494181    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqxv\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-kube-api-access-dmqxv\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.494248    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/579717b1-feef-4e2a-9da8-62721859adeb-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.494307    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/579717b1-feef-4e2a-9da8-62721859adeb-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.505453    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.507129    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/579717b1-feef-4e2a-9da8-62721859adeb-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.507374    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.508496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.507642    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.528662 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.520753    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.560551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.535570    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.560785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.553175    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmqxv\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-kube-api-access-dmqxv\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.560908 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.558949    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.578510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.574358    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/579717b1-feef-4e2a-9da8-62721859adeb-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.622840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.622806    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:49.734492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.733561    4282 generic.go:356] "Generic (PLEG): container finished" podID="c1c5b0d0-4a6e-4761-aa61-fddca0ec702f" containerID="8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812" exitCode=137
Oct 30 22:09:49.734492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.733714    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-2f1e5f"
Oct 30 22:09:49.734492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.734350    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c5b0d0-4a6e-4761-aa61-fddca0ec702f" path="/var/lib/kubelet/pods/c1c5b0d0-4a6e-4761-aa61-fddca0ec702f/volumes"
Oct 30 22:09:49.735906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.735245    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d159aeca-66aa-4b6b-b7cb-720375678e70" path="/var/lib/kubelet/pods/d159aeca-66aa-4b6b-b7cb-720375678e70/volumes"
Oct 30 22:09:49.777767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.777733    4282 scope.go:122] "RemoveContainer" containerID="8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812"
Oct 30 22:09:49.812452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.812427    4282 scope.go:122] "RemoveContainer" containerID="8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812"
Oct 30 22:09:49.812873 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:49.812845    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812\": container with ID starting with 8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812 not found: ID does not exist" containerID="8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812"
Oct 30 22:09:49.812940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:49.812885    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812"} err="failed to get container status \"8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812\": rpc error: code = NotFound desc = could not find container \"8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812\": container with ID starting with 8700effd821d21537b1e60e013a136a42f9cd077d919940b14821e7ff6e9b812 not found: ID does not exist"
Oct 30 22:09:50.073779 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.073743    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"]
Oct 30 22:09:50.076953 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:09:50.076922    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579717b1_feef_4e2a_9da8_62721859adeb.slice/crio-fb169fcff70dc9a7b1971e5f10d94383ccca9bccd497eb98c3cf124c51709057 WatchSource:0}: Error finding container fb169fcff70dc9a7b1971e5f10d94383ccca9bccd497eb98c3cf124c51709057: Status 404 returned error can't find the container with id fb169fcff70dc9a7b1971e5f10d94383ccca9bccd497eb98c3cf124c51709057
Oct 30 22:09:50.080341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.080302    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:09:50.080427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.080392    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:09:50.080484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.080430    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:09:50.742674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.742638    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" event={"ID":"579717b1-feef-4e2a-9da8-62721859adeb","Type":"ContainerStarted","Data":"8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d"}
Oct 30 22:09:50.743057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.742736    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" event={"ID":"579717b1-feef-4e2a-9da8-62721859adeb","Type":"ContainerStarted","Data":"fb169fcff70dc9a7b1971e5f10d94383ccca9bccd497eb98c3cf124c51709057"}
Oct 30 22:09:50.744798 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.744779    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-cacf24"
Oct 30 22:09:50.745320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.745143    4282 generic.go:356] "Generic (PLEG): container finished" podID="90712683-884e-45fb-a88b-79f48eb6a49a" containerID="6c5510cfbdc9cfb48ca2b05552dd8e00d79bc2ddfc579fa92b3340ab3f413787" exitCode=137
Oct 30 22:09:50.745320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.745200    4282 scope.go:122] "RemoveContainer" containerID="6c5510cfbdc9cfb48ca2b05552dd8e00d79bc2ddfc579fa92b3340ab3f413787"
Oct 30 22:09:50.766275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.766228    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podStartSLOduration=1.766213102 podStartE2EDuration="1.766213102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:50.764669908 +0000 UTC m=+1809.422766130" watchObservedRunningTime="2026-10-30 22:09:50.766213102 +0000 UTC m=+1809.424309325"
Oct 30 22:09:50.774855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.774832    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90712683-884e-45fb-a88b-79f48eb6a49a-kube-api-access-rktq8\" (UniqueName: \"kubernetes.io/projected/90712683-884e-45fb-a88b-79f48eb6a49a-kube-api-access-rktq8\") pod \"90712683-884e-45fb-a88b-79f48eb6a49a\" (UID: \"90712683-884e-45fb-a88b-79f48eb6a49a\") "
Oct 30 22:09:50.777048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.777023    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90712683-884e-45fb-a88b-79f48eb6a49a-kube-api-access-rktq8" pod "90712683-884e-45fb-a88b-79f48eb6a49a" (UID: "90712683-884e-45fb-a88b-79f48eb6a49a"). InnerVolumeSpecName "kube-api-access-rktq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:50.907491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:50.902490    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rktq8\" (UniqueName: \"kubernetes.io/projected/90712683-884e-45fb-a88b-79f48eb6a49a-kube-api-access-rktq8\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:51.467569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:51.467538    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90712683-884e-45fb-a88b-79f48eb6a49a" path="/var/lib/kubelet/pods/90712683-884e-45fb-a88b-79f48eb6a49a/volumes"
Oct 30 22:09:51.623301 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:51.623272    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:51.629111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:51.629088    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:51.753422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:51.753350    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7697-cacf24"
Oct 30 22:09:51.753887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:51.753690    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:51.754673 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:51.754654    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:09:53.874286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:53.874250    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7688-555213"]
Oct 30 22:09:53.884761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:53.884736    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-555213"]
Oct 30 22:09:53.885031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:53.885012    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-555213"
Oct 30 22:09:53.889654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:53.889516    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sfm7d\""
Oct 30 22:09:53.980382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:53.980352    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhnw\" (UniqueName: \"kubernetes.io/projected/f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645-kube-api-access-mnhnw\") pod \"test-curl-7688-555213\" (UID: \"f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645\") " pod="opendatahub/test-curl-7688-555213"
Oct 30 22:09:54.081837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.081803    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhnw\" (UniqueName: \"kubernetes.io/projected/f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645-kube-api-access-mnhnw\") pod \"test-curl-7688-555213\" (UID: \"f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645\") " pod="opendatahub/test-curl-7688-555213"
Oct 30 22:09:54.093611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.093587    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhnw\" (UniqueName: \"kubernetes.io/projected/f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645-kube-api-access-mnhnw\") pod \"test-curl-7688-555213\" (UID: \"f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645\") " pod="opendatahub/test-curl-7688-555213"
Oct 30 22:09:54.200215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.200194    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-555213"
Oct 30 22:09:54.484447 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.484416    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7688-555213"]
Oct 30 22:09:54.485391 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:09:54.485366    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1e4ed1d_9d6f_40d1_9bbb_6dfb6af51645.slice/crio-237bab7bf498d7d53b7d24e67073b6b785dbcccd87453d1f72174a2620a4cad8 WatchSource:0}: Error finding container 237bab7bf498d7d53b7d24e67073b6b785dbcccd87453d1f72174a2620a4cad8: Status 404 returned error can't find the container with id 237bab7bf498d7d53b7d24e67073b6b785dbcccd87453d1f72174a2620a4cad8
Oct 30 22:09:54.721804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.721770    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"]
Oct 30 22:09:54.726258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.726241    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:54.730987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.730955    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a39158\""
Oct 30 22:09:54.731328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.731305    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a39158-dockercfg-wrc2f\""
Oct 30 22:09:54.737392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.737369    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"]
Oct 30 22:09:54.771453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.771427    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"]
Oct 30 22:09:54.775777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.775757    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:54.789654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.789628    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"]
Oct 30 22:09:54.801986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.801957    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb62d9b-f7b2-4118-8669-cda1e51557e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96\" (UID: \"adb62d9b-f7b2-4118-8669-cda1e51557e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:54.802572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.802065    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqdq\" (UniqueName: \"kubernetes.io/projected/adb62d9b-f7b2-4118-8669-cda1e51557e6-kube-api-access-wkqdq\") pod \"payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96\" (UID: \"adb62d9b-f7b2-4118-8669-cda1e51557e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:54.919496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.915693    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8lq\" (UniqueName: \"kubernetes.io/projected/1aa65e87-cda6-468f-b669-44b5121ed498-kube-api-access-tj8lq\") pod \"payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz\" (UID: \"1aa65e87-cda6-468f-b669-44b5121ed498\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:54.919496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.915940    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb62d9b-f7b2-4118-8669-cda1e51557e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96\" (UID: \"adb62d9b-f7b2-4118-8669-cda1e51557e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:54.919496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.915995    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqdq\" (UniqueName: \"kubernetes.io/projected/adb62d9b-f7b2-4118-8669-cda1e51557e6-kube-api-access-wkqdq\") pod \"payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96\" (UID: \"adb62d9b-f7b2-4118-8669-cda1e51557e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:54.919496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.916161    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa65e87-cda6-468f-b669-44b5121ed498-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz\" (UID: \"1aa65e87-cda6-468f-b669-44b5121ed498\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:54.919496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.917181    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb62d9b-f7b2-4118-8669-cda1e51557e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96\" (UID: \"adb62d9b-f7b2-4118-8669-cda1e51557e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:54.997542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:54.995131    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:09:55.049505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.029717    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa65e87-cda6-468f-b669-44b5121ed498-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz\" (UID: \"1aa65e87-cda6-468f-b669-44b5121ed498\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:55.049505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.029908    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8lq\" (UniqueName: \"kubernetes.io/projected/1aa65e87-cda6-468f-b669-44b5121ed498-kube-api-access-tj8lq\") pod \"payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz\" (UID: \"1aa65e87-cda6-468f-b669-44b5121ed498\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:55.050082 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.050049    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa65e87-cda6-468f-b669-44b5121ed498-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz\" (UID: \"1aa65e87-cda6-468f-b669-44b5121ed498\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:55.128508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.127186    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-555213" event={"ID":"f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645","Type":"ContainerStarted","Data":"60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc"}
Oct 30 22:09:55.128508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.127261    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7688-555213" event={"ID":"f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645","Type":"ContainerStarted","Data":"237bab7bf498d7d53b7d24e67073b6b785dbcccd87453d1f72174a2620a4cad8"}
Oct 30 22:09:55.159051 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.156085    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8lq\" (UniqueName: \"kubernetes.io/projected/1aa65e87-cda6-468f-b669-44b5121ed498-kube-api-access-tj8lq\") pod \"payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz\" (UID: \"1aa65e87-cda6-468f-b669-44b5121ed498\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:55.167540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.167457    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7688-555213" podStartSLOduration=2.167437432 podStartE2EDuration="2.167437432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:55.15265014 +0000 UTC m=+1813.810746364" watchObservedRunningTime="2026-10-30 22:09:55.167437432 +0000 UTC m=+1813.825533656"
Oct 30 22:09:55.181268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.173773    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqdq\" (UniqueName: \"kubernetes.io/projected/adb62d9b-f7b2-4118-8669-cda1e51557e6-kube-api-access-wkqdq\") pod \"payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96\" (UID: \"adb62d9b-f7b2-4118-8669-cda1e51557e6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:55.346580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.345524    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:55.396289 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.396246    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:55.650506 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:09:55.650440    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb62d9b_f7b2_4118_8669_cda1e51557e6.slice/crio-19e856fcfd09e2aad76bbd691f5749439d2ae2d44919ffd8511a8228b3625c34 WatchSource:0}: Error finding container 19e856fcfd09e2aad76bbd691f5749439d2ae2d44919ffd8511a8228b3625c34: Status 404 returned error can't find the container with id 19e856fcfd09e2aad76bbd691f5749439d2ae2d44919ffd8511a8228b3625c34
Oct 30 22:09:55.651981 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.651956    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"]
Oct 30 22:09:55.726536 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:09:55.726505    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa65e87_cda6_468f_b669_44b5121ed498.slice/crio-555ef1d9c999094ebf18fc69d4b677f62cc6db4a0e6a93b85e2fb98a127ff038 WatchSource:0}: Error finding container 555ef1d9c999094ebf18fc69d4b677f62cc6db4a0e6a93b85e2fb98a127ff038: Status 404 returned error can't find the container with id 555ef1d9c999094ebf18fc69d4b677f62cc6db4a0e6a93b85e2fb98a127ff038
Oct 30 22:09:55.726661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:55.726595    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"]
Oct 30 22:09:56.212164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:56.212126    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz" event={"ID":"1aa65e87-cda6-468f-b669-44b5121ed498","Type":"ContainerStarted","Data":"555ef1d9c999094ebf18fc69d4b677f62cc6db4a0e6a93b85e2fb98a127ff038"}
Oct 30 22:09:56.219559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:56.219514    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96" event={"ID":"adb62d9b-f7b2-4118-8669-cda1e51557e6","Type":"ContainerStarted","Data":"1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7"}
Oct 30 22:09:56.219853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:56.219830    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96" event={"ID":"adb62d9b-f7b2-4118-8669-cda1e51557e6","Type":"ContainerStarted","Data":"19e856fcfd09e2aad76bbd691f5749439d2ae2d44919ffd8511a8228b3625c34"}
Oct 30 22:09:56.220022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:56.220009    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:09:56.908619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:56.908550    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96" podStartSLOduration=2.908531205 podStartE2EDuration="2.908531205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:56.250840115 +0000 UTC m=+1814.908936351" watchObservedRunningTime="2026-10-30 22:09:56.908531205 +0000 UTC m=+1815.566627431"
Oct 30 22:09:56.909899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:56.909870    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7688-555213"]
Oct 30 22:09:56.915949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:56.915924    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7688-555213"]
Oct 30 22:09:57.227383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:57.227342    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz" event={"ID":"1aa65e87-cda6-468f-b669-44b5121ed498","Type":"ContainerStarted","Data":"40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0"}
Oct 30 22:09:57.227898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:57.227698    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7688-555213" podUID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" containerName="test-curl-7688-555213" containerID="cri-o://60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc" gracePeriod=2
Oct 30 22:09:57.227898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:57.227745    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:09:57.245281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:57.245234    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz" podStartSLOduration=3.245223084 podStartE2EDuration="3.245223084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:57.244714074 +0000 UTC m=+1815.902810294" watchObservedRunningTime="2026-10-30 22:09:57.245223084 +0000 UTC m=+1815.903319306"
Oct 30 22:09:57.246578 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:09:57.246543    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7688-555213\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" pod="opendatahub/test-curl-7688-555213"
Oct 30 22:09:59.703237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:59.703209    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-555213"
Oct 30 22:09:59.745224 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:59.745191    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645-kube-api-access-mnhnw\" (UniqueName: \"kubernetes.io/projected/f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645-kube-api-access-mnhnw\") pod \"f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645\" (UID: \"f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645\") "
Oct 30 22:09:59.747639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:59.747612    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645-kube-api-access-mnhnw" pod "f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" (UID: "f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645"). InnerVolumeSpecName "kube-api-access-mnhnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:59.847935 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:09:59.847853    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhnw\" (UniqueName: \"kubernetes.io/projected/f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645-kube-api-access-mnhnw\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:00.326650 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:00.326609    4282 generic.go:356] "Generic (PLEG): container finished" podID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" containerID="60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc" exitCode=137
Oct 30 22:10:00.326846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:00.326663    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7688-555213"
Oct 30 22:10:00.326846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:00.326718    4282 scope.go:122] "RemoveContainer" containerID="60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc"
Oct 30 22:10:00.338388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:00.338371    4282 scope.go:122] "RemoveContainer" containerID="60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc"
Oct 30 22:10:00.338734 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:00.338713    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc\": container with ID starting with 60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc not found: ID does not exist" containerID="60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc"
Oct 30 22:10:00.338800 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:00.338743    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc"} err="failed to get container status \"60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc\": rpc error: code = NotFound desc = could not find container \"60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc\": container with ID starting with 60e400d99d5ddf011ea0045c70b7f63253a7af0b451384dfdd42be7e2ecab0fc not found: ID does not exist"
Oct 30 22:10:00.339376 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:00.339358    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7688-555213\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" pod="opendatahub/test-curl-7688-555213"
Oct 30 22:10:01.464858 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:01.464822    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7688-555213\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" pod="opendatahub/test-curl-7688-555213"
Oct 30 22:10:01.468260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:01.468223    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" path="/var/lib/kubelet/pods/f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645/volumes"
Oct 30 22:10:03.953006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:03.952964    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"]
Oct 30 22:10:03.953892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:03.953870    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" containerName="test-curl-7688-555213"
Oct 30 22:10:03.953892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:03.953892    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" containerName="test-curl-7688-555213"
Oct 30 22:10:03.954141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:03.954127    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1e4ed1d-9d6f-40d1-9bbb-6dfb6af51645" containerName="test-curl-7688-555213"
Oct 30 22:10:03.957775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:03.957756    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:03.965266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:03.965244    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1a5bdf\""
Oct 30 22:10:03.971809 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:03.971785    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"]
Oct 30 22:10:04.007193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.007160    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1a5bdf-dockercfg-65kxm\""
Oct 30 22:10:04.010547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.010518    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"]
Oct 30 22:10:04.014943 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.014923    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:04.030734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.027538    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"]
Oct 30 22:10:04.041424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.041400    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twr4k\" (UniqueName: \"kubernetes.io/projected/07b0d53f-2a75-460e-89f9-a8c2a03608de-kube-api-access-twr4k\") pod \"payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29\" (UID: \"07b0d53f-2a75-460e-89f9-a8c2a03608de\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:04.041729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.041713    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0d53f-2a75-460e-89f9-a8c2a03608de-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29\" (UID: \"07b0d53f-2a75-460e-89f9-a8c2a03608de\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:04.201557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.194596    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0d53f-2a75-460e-89f9-a8c2a03608de-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29\" (UID: \"07b0d53f-2a75-460e-89f9-a8c2a03608de\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:04.232763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.232677    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twr4k\" (UniqueName: \"kubernetes.io/projected/07b0d53f-2a75-460e-89f9-a8c2a03608de-kube-api-access-twr4k\") pod \"payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29\" (UID: \"07b0d53f-2a75-460e-89f9-a8c2a03608de\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:04.258498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.233204    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljcnv\" (UniqueName: \"kubernetes.io/projected/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-kube-api-access-ljcnv\") pod \"payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55\" (UID: \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:04.258905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.258875    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55\" (UID: \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:04.285494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.229500    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0d53f-2a75-460e-89f9-a8c2a03608de-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29\" (UID: \"07b0d53f-2a75-460e-89f9-a8c2a03608de\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:04.392549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.363296    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljcnv\" (UniqueName: \"kubernetes.io/projected/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-kube-api-access-ljcnv\") pod \"payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55\" (UID: \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:04.607784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.607662    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55\" (UID: \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:04.630621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.539231    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljcnv\" (UniqueName: \"kubernetes.io/projected/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-kube-api-access-ljcnv\") pod \"payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55\" (UID: \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:04.630621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.584222    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"]
Oct 30 22:10:04.630621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.416643    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twr4k\" (UniqueName: \"kubernetes.io/projected/07b0d53f-2a75-460e-89f9-a8c2a03608de-kube-api-access-twr4k\") pod \"payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29\" (UID: \"07b0d53f-2a75-460e-89f9-a8c2a03608de\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:04.630621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.610385    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55\" (UID: \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:04.641795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.641751    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:04.692572 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:04.689173    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:10:04.937582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:04.875083    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:05.261494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.261419    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"]
Oct 30 22:10:05.266820 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:05.266792    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc88f186_1a26_4dde_a4b3_2cda4e3f218e.slice/crio-092ce6eb7892c3f6110cf2f295541d622f910774cbd0fe18a2e39488c03176c2 WatchSource:0}: Error finding container 092ce6eb7892c3f6110cf2f295541d622f910774cbd0fe18a2e39488c03176c2: Status 404 returned error can't find the container with id 092ce6eb7892c3f6110cf2f295541d622f910774cbd0fe18a2e39488c03176c2
Oct 30 22:10:05.323781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.323755    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"]
Oct 30 22:10:05.336977 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:05.336950    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b0d53f_2a75_460e_89f9_a8c2a03608de.slice/crio-0e7f5a7eaa8ea6da006843d9b062e75e9dee08efe27a74733bd1a9d327a7dfc1 WatchSource:0}: Error finding container 0e7f5a7eaa8ea6da006843d9b062e75e9dee08efe27a74733bd1a9d327a7dfc1: Status 404 returned error can't find the container with id 0e7f5a7eaa8ea6da006843d9b062e75e9dee08efe27a74733bd1a9d327a7dfc1
Oct 30 22:10:05.600602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.600503    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55" event={"ID":"cc88f186-1a26-4dde-a4b3-2cda4e3f218e","Type":"ContainerStarted","Data":"27fbf220cfdee75eb2802990bbb01615208f8c53d7eb13a55ff9ecc721edf5ef"}
Oct 30 22:10:05.600602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.600562    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55" event={"ID":"cc88f186-1a26-4dde-a4b3-2cda4e3f218e","Type":"ContainerStarted","Data":"092ce6eb7892c3f6110cf2f295541d622f910774cbd0fe18a2e39488c03176c2"}
Oct 30 22:10:05.600854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.600736    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:05.603126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.603097    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29" event={"ID":"07b0d53f-2a75-460e-89f9-a8c2a03608de","Type":"ContainerStarted","Data":"73b4828efe51643513de9e09b8a4b24523559fe30af62b773740c7258133ffda"}
Oct 30 22:10:05.603277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.603152    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29" event={"ID":"07b0d53f-2a75-460e-89f9-a8c2a03608de","Type":"ContainerStarted","Data":"0e7f5a7eaa8ea6da006843d9b062e75e9dee08efe27a74733bd1a9d327a7dfc1"}
Oct 30 22:10:05.603277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.603181    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:05.619045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.619002    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55" podStartSLOduration=2.618988091 podStartE2EDuration="2.618988091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:05.61723668 +0000 UTC m=+1824.275332904" watchObservedRunningTime="2026-10-30 22:10:05.618988091 +0000 UTC m=+1824.277084360"
Oct 30 22:10:05.638388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:05.638343    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29" podStartSLOduration=2.638331027 podStartE2EDuration="2.638331027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:05.634594483 +0000 UTC m=+1824.292690706" watchObservedRunningTime="2026-10-30 22:10:05.638331027 +0000 UTC m=+1824.296427290"
Oct 30 22:10:08.237856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:08.237830    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:10:08.238282 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:08.237894    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:10:13.386622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:13.386585    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"]
Oct 30 22:10:15.379771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.379732    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"]
Oct 30 22:10:15.386828 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.386806    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.392121 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.392097    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bacea0-openshift-default-dockercfg-f9dbz\""
Oct 30 22:10:15.399924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.399753    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"]
Oct 30 22:10:15.449778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.449744    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-certs\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.450025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.449848    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-data\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.450025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.449889    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-token\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.450025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.449948    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-socket\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.450215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.450140    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-credential-socket\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.450215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.450175    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-envoy\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.450326 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.450264    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wpsq\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-kube-api-access-8wpsq\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.450326 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.450292    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e713fa3-0235-4cb6-aea0-65fa0591293d-istiod-ca-cert\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.450434 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.450325    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-podinfo\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.553918    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wpsq\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-kube-api-access-8wpsq\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.553984    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e713fa3-0235-4cb6-aea0-65fa0591293d-istiod-ca-cert\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.554037    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-podinfo\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.554153    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-certs\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.554303    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-data\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.554361    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-token\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.554441    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-socket\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.554742    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-credential-socket\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.554793    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-envoy\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.556489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.556299    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-data\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.568493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.564625    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-socket\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.568493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.564936    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-credential-socket\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.583537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.574034    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-certs\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.583537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.557133    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e713fa3-0235-4cb6-aea0-65fa0591293d-istiod-ca-cert\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.587490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.587260    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-envoy\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.603489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.596908    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-token\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.613565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.610582    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-podinfo\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.667348 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.667319    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wpsq\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-kube-api-access-8wpsq\") pod \"e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") " pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:15.719876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:15.710881    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:16.237949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:16.237912    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"]
Oct 30 22:10:16.238069 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:16.237919    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e713fa3_0235_4cb6_aea0_65fa0591293d.slice/crio-c3b72cde4f8334bb3574831a1789084bdeb18f21477e968476dbff40d1da6fd0 WatchSource:0}: Error finding container c3b72cde4f8334bb3574831a1789084bdeb18f21477e968476dbff40d1da6fd0: Status 404 returned error can't find the container with id c3b72cde4f8334bb3574831a1789084bdeb18f21477e968476dbff40d1da6fd0
Oct 30 22:10:16.241277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:16.241237    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:16.241364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:16.241322    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:16.241404 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:16.241363    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:16.637433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:16.635963    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:10:16.646574 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:16.646544    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:10:16.775881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:16.775839    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" event={"ID":"3e713fa3-0235-4cb6-aea0-65fa0591293d","Type":"ContainerStarted","Data":"df7e3128ba0207a81ce8384143609033ee1230c7855c8d0b1a0cab08bd18d311"}
Oct 30 22:10:16.776057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:16.775898    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" event={"ID":"3e713fa3-0235-4cb6-aea0-65fa0591293d","Type":"ContainerStarted","Data":"c3b72cde4f8334bb3574831a1789084bdeb18f21477e968476dbff40d1da6fd0"}
Oct 30 22:10:17.712826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:17.712795    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:17.718956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:17.718932    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:17.743299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:17.743244    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podStartSLOduration=2.743230763 podStartE2EDuration="2.743230763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:16.802423961 +0000 UTC m=+1835.460520176" watchObservedRunningTime="2026-10-30 22:10:17.743230763 +0000 UTC m=+1836.401326985"
Oct 30 22:10:17.792244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:17.792183    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:17.793447 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:17.793424    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:10:19.672794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:19.672753    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4" podUID="c266b897-204e-44bb-94ba-1e20c7013b5c" containerName="main" containerID="cri-o://6054a8069dcc44b269a0aa71845c88d80b824ccc8e6dbd140d41e240b6c36567" gracePeriod=45
Oct 30 22:10:19.809053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:19.809007    4282 generic.go:356] "Generic (PLEG): container finished" podID="c266b897-204e-44bb-94ba-1e20c7013b5c" containerID="6054a8069dcc44b269a0aa71845c88d80b824ccc8e6dbd140d41e240b6c36567" exitCode=0
Oct 30 22:10:19.809201 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:19.809071    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4" event={"ID":"c266b897-204e-44bb-94ba-1e20c7013b5c","Type":"ContainerDied","Data":"6054a8069dcc44b269a0aa71845c88d80b824ccc8e6dbd140d41e240b6c36567"}
Oct 30 22:10:19.957690 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:19.957662    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:10:20.083020 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.082983    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c266b897-204e-44bb-94ba-1e20c7013b5c-kube-api-access-s4mbf\" (UniqueName: \"kubernetes.io/projected/c266b897-204e-44bb-94ba-1e20c7013b5c-kube-api-access-s4mbf\") pod \"c266b897-204e-44bb-94ba-1e20c7013b5c\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") "
Oct 30 22:10:20.083225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.083036    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-tmp-dir\") pod \"c266b897-204e-44bb-94ba-1e20c7013b5c\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") "
Oct 30 22:10:20.083225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.083191    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-home\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-home\") pod \"c266b897-204e-44bb-94ba-1e20c7013b5c\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") "
Oct 30 22:10:20.083349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.083301    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c266b897-204e-44bb-94ba-1e20c7013b5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/c266b897-204e-44bb-94ba-1e20c7013b5c-tls-certs\") pod \"c266b897-204e-44bb-94ba-1e20c7013b5c\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") "
Oct 30 22:10:20.083349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.083333    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-model-cache\") pod \"c266b897-204e-44bb-94ba-1e20c7013b5c\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") "
Oct 30 22:10:20.083454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.083296    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-tmp-dir" pod "c266b897-204e-44bb-94ba-1e20c7013b5c" (UID: "c266b897-204e-44bb-94ba-1e20c7013b5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:20.083454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.083427    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-dshm\") pod \"c266b897-204e-44bb-94ba-1e20c7013b5c\" (UID: \"c266b897-204e-44bb-94ba-1e20c7013b5c\") "
Oct 30 22:10:20.083616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.083453    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-home" pod "c266b897-204e-44bb-94ba-1e20c7013b5c" (UID: "c266b897-204e-44bb-94ba-1e20c7013b5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:20.083668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.083646    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-model-cache" pod "c266b897-204e-44bb-94ba-1e20c7013b5c" (UID: "c266b897-204e-44bb-94ba-1e20c7013b5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:20.084155 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.084134    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.084248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.084159    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.084248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.084172    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.085726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.085705    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-dshm" pod "c266b897-204e-44bb-94ba-1e20c7013b5c" (UID: "c266b897-204e-44bb-94ba-1e20c7013b5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:20.085984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.085961    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c266b897-204e-44bb-94ba-1e20c7013b5c-tls-certs" pod "c266b897-204e-44bb-94ba-1e20c7013b5c" (UID: "c266b897-204e-44bb-94ba-1e20c7013b5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:20.086094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.086003    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c266b897-204e-44bb-94ba-1e20c7013b5c-kube-api-access-s4mbf" pod "c266b897-204e-44bb-94ba-1e20c7013b5c" (UID: "c266b897-204e-44bb-94ba-1e20c7013b5c"). InnerVolumeSpecName "kube-api-access-s4mbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:20.189548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.185842    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c266b897-204e-44bb-94ba-1e20c7013b5c-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.189548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.185877    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c266b897-204e-44bb-94ba-1e20c7013b5c-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.189548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.185894    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4mbf\" (UniqueName: \"kubernetes.io/projected/c266b897-204e-44bb-94ba-1e20c7013b5c-kube-api-access-s4mbf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.620542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.601558    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"]
Oct 30 22:10:20.620542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.602575    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c266b897-204e-44bb-94ba-1e20c7013b5c" containerName="main"
Oct 30 22:10:20.620542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.602592    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c266b897-204e-44bb-94ba-1e20c7013b5c" containerName="main"
Oct 30 22:10:20.620542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.602796    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c266b897-204e-44bb-94ba-1e20c7013b5c" containerName="main"
Oct 30 22:10:20.620542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.607224    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:20.630595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.627696    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-bacea0\""
Oct 30 22:10:20.630595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.630335    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-bacea0-dockercfg-dp7d4\""
Oct 30 22:10:20.632648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.632621    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"]
Oct 30 22:10:20.661977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.661942    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"]
Oct 30 22:10:20.668705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.668670    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.668858 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.668825    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"]
Oct 30 22:10:20.673663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.673640    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:20.692683 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.690288    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"]
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.695549    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x\" (UID: \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.695633    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d838db-a10d-4705-8c50-09e5ce26689a-istiod-ca-cert\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.695715    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-certs\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.695767    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d838db-a10d-4705-8c50-09e5ce26689a-istio-podinfo\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.695857    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rckjj\" (UniqueName: \"kubernetes.io/projected/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-kube-api-access-rckjj\") pod \"payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x\" (UID: \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.695899    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-data\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.695925    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndwd\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-kube-api-access-dndwd\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.695958    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-envoy\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.696116    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-istio-token\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.696174    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-credential-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.697094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.696232    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.707422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.707394    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"]
Oct 30 22:10:20.730942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.730915    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-26a128-openshift-default-dockercfg-7xjnf\""
Oct 30 22:10:20.797871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.797836    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln\" (UID: \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:20.798056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.797891    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-istio-token\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.797986    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-credential-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798048    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798157    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x\" (UID: \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:20.798235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798201    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8vcf\" (UniqueName: \"kubernetes.io/projected/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-kube-api-access-s8vcf\") pod \"payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln\" (UID: \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:20.798366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798277    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d838db-a10d-4705-8c50-09e5ce26689a-istiod-ca-cert\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798356    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-certs\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798434 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798412    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d838db-a10d-4705-8c50-09e5ce26689a-istio-podinfo\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798486 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798445    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-credential-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798486 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798456    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-socket\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798803 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798763    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-certs\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.798933 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798906    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckjj\" (UniqueName: \"kubernetes.io/projected/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-kube-api-access-rckjj\") pod \"payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x\" (UID: \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:20.798997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.798968    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-data\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.799057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.799000    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndwd\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-kube-api-access-dndwd\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.799057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.799033    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-envoy\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.799216 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.799094    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x\" (UID: \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:20.799405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.799253    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d838db-a10d-4705-8c50-09e5ce26689a-istiod-ca-cert\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.799405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.799276    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-data\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.801729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.801698    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d838db-a10d-4705-8c50-09e5ce26689a-istio-podinfo\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.801910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.801887    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-envoy\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.812570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.812545    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-istio-token\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.814306 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.814286    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndwd\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-kube-api-access-dndwd\") pod \"e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") " pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:20.816371 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.816351    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckjj\" (UniqueName: \"kubernetes.io/projected/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-kube-api-access-rckjj\") pod \"payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x\" (UID: \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:20.820162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.820144    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"
Oct 30 22:10:20.820257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.820182    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4" event={"ID":"c266b897-204e-44bb-94ba-1e20c7013b5c","Type":"ContainerDied","Data":"274e03de42f3a9f801c7365513c3a8640f3b4de414d0bf0f63b41b0d9ac7fbfd"}
Oct 30 22:10:20.820257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.820232    4282 scope.go:122] "RemoveContainer" containerID="6054a8069dcc44b269a0aa71845c88d80b824ccc8e6dbd140d41e240b6c36567"
Oct 30 22:10:20.934492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.923735    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln\" (UID: \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:20.934492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.923936    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8vcf\" (UniqueName: \"kubernetes.io/projected/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-kube-api-access-s8vcf\") pod \"payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln\" (UID: \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:20.938552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.936262    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln\" (UID: \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:20.967367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.965535    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"]
Oct 30 22:10:20.967367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.965825    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:21.007170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:20.994718    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8vcf\" (UniqueName: \"kubernetes.io/projected/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-kube-api-access-s8vcf\") pod \"payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln\" (UID: \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:21.007170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.006847    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:21.028559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.021843    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:21.062565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.057451    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-ecfbd6/auth-test-model-ecfbd6-kserve-59b7bcbd4f-mkct4"]
Oct 30 22:10:21.483373 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.483335    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c266b897-204e-44bb-94ba-1e20c7013b5c" path="/var/lib/kubelet/pods/c266b897-204e-44bb-94ba-1e20c7013b5c/volumes"
Oct 30 22:10:21.781186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.781161    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"]
Oct 30 22:10:21.782529 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:21.782502    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7568c7b_2b8f_41d2_8554_e33f16cbf42f.slice/crio-aa44b0215827c3672695ed261b478c37de559f332811fc31653e5fc7b71f7fea WatchSource:0}: Error finding container aa44b0215827c3672695ed261b478c37de559f332811fc31653e5fc7b71f7fea: Status 404 returned error can't find the container with id aa44b0215827c3672695ed261b478c37de559f332811fc31653e5fc7b71f7fea
Oct 30 22:10:21.815308 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:21.815278    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf37c164_c4c7_4e73_b615_bb1b9ae62dea.slice/crio-d0c53149e8872c17e189c569639f1f524581126b10d667be1c1a51e6855ec7ed WatchSource:0}: Error finding container d0c53149e8872c17e189c569639f1f524581126b10d667be1c1a51e6855ec7ed: Status 404 returned error can't find the container with id d0c53149e8872c17e189c569639f1f524581126b10d667be1c1a51e6855ec7ed
Oct 30 22:10:21.818391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.818365    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"]
Oct 30 22:10:21.824518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.824442    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"]
Oct 30 22:10:21.825613 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.825575    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:21.825726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.825645    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:21.825726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.825691    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:21.832505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.832447    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln" event={"ID":"bf37c164-c4c7-4e73-b615-bb1b9ae62dea","Type":"ContainerStarted","Data":"d0c53149e8872c17e189c569639f1f524581126b10d667be1c1a51e6855ec7ed"}
Oct 30 22:10:21.835098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:21.835074    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x" event={"ID":"f7568c7b-2b8f-41d2-8554-e33f16cbf42f","Type":"ContainerStarted","Data":"aa44b0215827c3672695ed261b478c37de559f332811fc31653e5fc7b71f7fea"}
Oct 30 22:10:22.843758 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.843721    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln" event={"ID":"bf37c164-c4c7-4e73-b615-bb1b9ae62dea","Type":"ContainerStarted","Data":"46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935"}
Oct 30 22:10:22.844253 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.843794    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:22.845523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.845497    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x" event={"ID":"f7568c7b-2b8f-41d2-8554-e33f16cbf42f","Type":"ContainerStarted","Data":"6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808"}
Oct 30 22:10:22.845674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.845542    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:22.847276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.847253    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" event={"ID":"71d838db-a10d-4705-8c50-09e5ce26689a","Type":"ContainerStarted","Data":"92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f"}
Oct 30 22:10:22.847397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.847291    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" event={"ID":"71d838db-a10d-4705-8c50-09e5ce26689a","Type":"ContainerStarted","Data":"c2148ca1dd9598226578c98c2833ec2d4f013e998338aa00e3d7b93d7cb8e2c5"}
Oct 30 22:10:22.867482 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.867423    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln" podStartSLOduration=2.867408674 podStartE2EDuration="2.867408674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:22.86158191 +0000 UTC m=+1841.519678135" watchObservedRunningTime="2026-10-30 22:10:22.867408674 +0000 UTC m=+1841.525504899"
Oct 30 22:10:22.885400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.885346    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podStartSLOduration=2.885330885 podStartE2EDuration="2.885330885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:22.882214666 +0000 UTC m=+1841.540310888" watchObservedRunningTime="2026-10-30 22:10:22.885330885 +0000 UTC m=+1841.543427109"
Oct 30 22:10:22.904096 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:22.904049    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x" podStartSLOduration=2.904037158 podStartE2EDuration="2.904037158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:22.899337236 +0000 UTC m=+1841.557433496" watchObservedRunningTime="2026-10-30 22:10:22.904037158 +0000 UTC m=+1841.562133381"
Oct 30 22:10:23.008157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:23.008075    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:23.028940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:23.028908    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:23.228384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:23.228357    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"]
Oct 30 22:10:23.927546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:23.927513    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:23.933674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:23.933647    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:10:26.687985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.687953    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"]
Oct 30 22:10:26.710384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.704616    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:26.710384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.709174    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-26a128-dockercfg-kj5cv\""
Oct 30 22:10:26.721608 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.721204    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"]
Oct 30 22:10:26.739799 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.739778    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-26a128\""
Oct 30 22:10:26.838496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.812648    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1503756-ef8b-4706-bdb1-d11f3dda8b56-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl\" (UID: \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:26.868495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.840664    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkphz\" (UniqueName: \"kubernetes.io/projected/b1503756-ef8b-4706-bdb1-d11f3dda8b56-kube-api-access-rkphz\") pod \"payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl\" (UID: \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:26.990232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.990130    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkphz\" (UniqueName: \"kubernetes.io/projected/b1503756-ef8b-4706-bdb1-d11f3dda8b56-kube-api-access-rkphz\") pod \"payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl\" (UID: \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:26.990828 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:26.990802    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1503756-ef8b-4706-bdb1-d11f3dda8b56-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl\" (UID: \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:27.009177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:27.009091    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1503756-ef8b-4706-bdb1-d11f3dda8b56-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl\" (UID: \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:27.076735 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:27.076700    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkphz\" (UniqueName: \"kubernetes.io/projected/b1503756-ef8b-4706-bdb1-d11f3dda8b56-kube-api-access-rkphz\") pod \"payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl\" (UID: \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:27.328457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:27.328365    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:27.635243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:27.635213    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"]
Oct 30 22:10:27.637169 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:27.637140    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1503756_ef8b_4706_bdb1_d11f3dda8b56.slice/crio-6f4f1410c82f95faf75fbbc4e33b97676a2c390c2caac4b54017531e1d22243b WatchSource:0}: Error finding container 6f4f1410c82f95faf75fbbc4e33b97676a2c390c2caac4b54017531e1d22243b: Status 404 returned error can't find the container with id 6f4f1410c82f95faf75fbbc4e33b97676a2c390c2caac4b54017531e1d22243b
Oct 30 22:10:28.044351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:28.044317    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl" event={"ID":"b1503756-ef8b-4706-bdb1-d11f3dda8b56","Type":"ContainerStarted","Data":"6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a"}
Oct 30 22:10:28.044351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:28.044367    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl" event={"ID":"b1503756-ef8b-4706-bdb1-d11f3dda8b56","Type":"ContainerStarted","Data":"6f4f1410c82f95faf75fbbc4e33b97676a2c390c2caac4b54017531e1d22243b"}
Oct 30 22:10:28.044919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:28.044397    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:28.073584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:28.073420    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl" podStartSLOduration=2.073406505 podStartE2EDuration="2.073406505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:28.071281753 +0000 UTC m=+1846.729377970" watchObservedRunningTime="2026-10-30 22:10:28.073406505 +0000 UTC m=+1846.731502728"
Oct 30 22:10:28.556912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:28.552094    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb" podUID="fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" containerName="main" containerID="cri-o://c9c89cbf0e5bcbe4bbfad871645e33ca46775784434cd827d56bc9790fe55858" gracePeriod=45
Oct 30 22:10:28.655452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:28.655423    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"]
Oct 30 22:10:28.696597 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:28.696565    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb" podUID="fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.176:8000/ready\": dial tcp 10.133.0.176:8000: connect: connection refused"
Oct 30 22:10:29.055779 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.055748    4282 generic.go:356] "Generic (PLEG): container finished" podID="fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" containerID="c9c89cbf0e5bcbe4bbfad871645e33ca46775784434cd827d56bc9790fe55858" exitCode=0
Oct 30 22:10:29.056221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.055841    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb" event={"ID":"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4","Type":"ContainerDied","Data":"c9c89cbf0e5bcbe4bbfad871645e33ca46775784434cd827d56bc9790fe55858"}
Oct 30 22:10:29.056221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.055896    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb" event={"ID":"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4","Type":"ContainerDied","Data":"ac548b41418a9006d3d20fef783976051fc95c7af65eb5e2f6c57121514fb040"}
Oct 30 22:10:29.056221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.055913    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac548b41418a9006d3d20fef783976051fc95c7af65eb5e2f6c57121514fb040"
Oct 30 22:10:29.074952 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.074928    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:10:29.250496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.191843    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-home\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-home\") pod \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") "
Oct 30 22:10:29.308546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.304589    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tls-certs\") pod \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") "
Oct 30 22:10:29.308546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.304883    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-kube-api-access-gglzr\" (UniqueName: \"kubernetes.io/projected/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-kube-api-access-gglzr\") pod \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") "
Oct 30 22:10:29.308546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.305145    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tmp-dir\") pod \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") "
Oct 30 22:10:29.308546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.305173    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-dshm\") pod \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") "
Oct 30 22:10:29.308546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.305204    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-model-cache\") pod \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\" (UID: \"fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4\") "
Oct 30 22:10:29.308546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.230605    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-home" pod "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" (UID: "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:29.359772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.359659    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tmp-dir" pod "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" (UID: "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:29.400774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.393571    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-model-cache" pod "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" (UID: "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:29.432722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.429669    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-dshm" pod "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" (UID: "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:29.433044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.432990    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tls-certs" pod "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" (UID: "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:29.455517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.437392    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:29.455517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.437425    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:29.455517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.437440    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:29.455517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.437456    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:29.455517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.437488    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:29.681167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.681124    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-kube-api-access-gglzr" pod "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" (UID: "fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4"). InnerVolumeSpecName "kube-api-access-gglzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:29.780494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:29.775056    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gglzr\" (UniqueName: \"kubernetes.io/projected/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4-kube-api-access-gglzr\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:30.082492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:30.082108    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"
Oct 30 22:10:30.274498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:30.265376    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"]
Oct 30 22:10:30.344555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:30.305348    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0a2577/auth-test-model-0a2577-kserve-556bdf4fb9-kfctb"]
Oct 30 22:10:30.452238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:30.451597    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:30.633251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:30.632588    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz" podUID="0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.175:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:10:31.216080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.214862    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"]
Oct 30 22:10:31.216080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.216067    4282 cpu_manager.go:414] "containerMap: removing container" podUID="fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" containerName="main"
Oct 30 22:10:31.216080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.216087    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" containerName="main"
Oct 30 22:10:31.216775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.216352    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" containerName="main"
Oct 30 22:10:31.222503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.222452    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.231122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.230792    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-b9d868-openshift-default-dockercfg-rxwh7\""
Oct 30 22:10:31.239522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.236769    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"]
Oct 30 22:10:31.376964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.376905    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd34ab38-6807-4efc-961f-6b10adb0f268-istiod-ca-cert\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.377285 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.377250    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-envoy\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.377408 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.377386    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-podinfo\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.377567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.377551    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-data\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.377735 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.377720    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-token\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.377858 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.377841    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-certs\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.378030 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.378005    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-credential-socket\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.378217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.378192    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dkrk\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-kube-api-access-6dkrk\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.378438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.378410    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-socket\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.471307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.471234    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4" path="/var/lib/kubelet/pods/fc185ae6-ec6b-4117-8bdb-d88f8fbf2ae4/volumes"
Oct 30 22:10:31.479780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.479754    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd34ab38-6807-4efc-961f-6b10adb0f268-istiod-ca-cert\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.479907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.479802    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-envoy\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.479907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.479827    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-podinfo\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.479907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.479899    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-data\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.479959    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-token\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.479991    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-certs\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.480045    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-credential-socket\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.480113    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dkrk\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-kube-api-access-6dkrk\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.480149    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-socket\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480348 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.480295    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-data\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.480388    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-certs\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480608 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.480583    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-socket\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.480797    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-credential-socket\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.480876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.480827    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd34ab38-6807-4efc-961f-6b10adb0f268-istiod-ca-cert\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.482307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.482290    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-envoy\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.482558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.482537    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-podinfo\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.491383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.491360    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dkrk\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-kube-api-access-6dkrk\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.491506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.491379    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-token\") pod \"e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") " pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:31.550550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:31.550528    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:32.076082 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.076041    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"]
Oct 30 22:10:32.077669 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:32.077627    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd34ab38_6807_4efc_961f_6b10adb0f268.slice/crio-41ab5f335d95e782ae76a5a98b3e738fa466ee4ce79d4bf6497020c6697ba45f WatchSource:0}: Error finding container 41ab5f335d95e782ae76a5a98b3e738fa466ee4ce79d4bf6497020c6697ba45f: Status 404 returned error can't find the container with id 41ab5f335d95e782ae76a5a98b3e738fa466ee4ce79d4bf6497020c6697ba45f
Oct 30 22:10:32.081186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.081138    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:32.081284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.081233    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:32.081284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.081276    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:32.099624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.099599    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" event={"ID":"bd34ab38-6807-4efc-961f-6b10adb0f268","Type":"ContainerStarted","Data":"41ab5f335d95e782ae76a5a98b3e738fa466ee4ce79d4bf6497020c6697ba45f"}
Oct 30 22:10:32.344119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.344041    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7691-bbd583"]
Oct 30 22:10:32.348960 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.348808    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:32.354205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.354164    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sfm7d\""
Oct 30 22:10:32.356530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.356505    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7691-bbd583"]
Oct 30 22:10:32.414793 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.414695    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrb4\" (UniqueName: \"kubernetes.io/projected/8f29a413-6e8a-4a20-ac54-6d8f8b587624-kube-api-access-mrrb4\") pod \"test-curl-7691-bbd583\" (UID: \"8f29a413-6e8a-4a20-ac54-6d8f8b587624\") " pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:32.518489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.516924    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrb4\" (UniqueName: \"kubernetes.io/projected/8f29a413-6e8a-4a20-ac54-6d8f8b587624-kube-api-access-mrrb4\") pod \"test-curl-7691-bbd583\" (UID: \"8f29a413-6e8a-4a20-ac54-6d8f8b587624\") " pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:32.552594 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.552519    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrb4\" (UniqueName: \"kubernetes.io/projected/8f29a413-6e8a-4a20-ac54-6d8f8b587624-kube-api-access-mrrb4\") pod \"test-curl-7691-bbd583\" (UID: \"8f29a413-6e8a-4a20-ac54-6d8f8b587624\") " pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:32.677084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.677054    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:32.965773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:32.965746    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7691-bbd583"]
Oct 30 22:10:32.966306 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:32.966281    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f29a413_6e8a_4a20_ac54_6d8f8b587624.slice/crio-e2287977105745a0454d728933eb91ba4d5d197119423ee1965b74d72eee494c WatchSource:0}: Error finding container e2287977105745a0454d728933eb91ba4d5d197119423ee1965b74d72eee494c: Status 404 returned error can't find the container with id e2287977105745a0454d728933eb91ba4d5d197119423ee1965b74d72eee494c
Oct 30 22:10:33.111785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.111752    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7691-bbd583" event={"ID":"8f29a413-6e8a-4a20-ac54-6d8f8b587624","Type":"ContainerStarted","Data":"bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5"}
Oct 30 22:10:33.111982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.111798    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7691-bbd583" event={"ID":"8f29a413-6e8a-4a20-ac54-6d8f8b587624","Type":"ContainerStarted","Data":"e2287977105745a0454d728933eb91ba4d5d197119423ee1965b74d72eee494c"}
Oct 30 22:10:33.113752 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.113725    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" event={"ID":"bd34ab38-6807-4efc-961f-6b10adb0f268","Type":"ContainerStarted","Data":"4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d"}
Oct 30 22:10:33.131058 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.131014    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7691-bbd583" podStartSLOduration=1.131001838 podStartE2EDuration="1.131001838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:33.12613096 +0000 UTC m=+1851.784227183" watchObservedRunningTime="2026-10-30 22:10:33.131001838 +0000 UTC m=+1851.789098062"
Oct 30 22:10:33.148375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.148316    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podStartSLOduration=2.148302421 podStartE2EDuration="2.148302421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:33.147214935 +0000 UTC m=+1851.805311160" watchObservedRunningTime="2026-10-30 22:10:33.148302421 +0000 UTC m=+1851.806398643"
Oct 30 22:10:33.551658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.551624    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:33.557983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.557957    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:33.930643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.930617    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:10:33.930892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:33.930680    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:10:34.024197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:34.024160    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7691-bbd583"]
Oct 30 22:10:34.042355 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:34.042325    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7691-bbd583"]
Oct 30 22:10:34.210296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:34.210215    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:34.211207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:34.211186    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:10:34.664141 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:34.664055    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:10:35.274696 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:35.274621    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7691-bbd583" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" containerName="test-curl-7691-bbd583" containerID="cri-o://bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5" gracePeriod=2
Oct 30 22:10:35.313062 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:35.277099    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:37.598664 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:37.598638    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:37.681990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:37.681961    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f29a413-6e8a-4a20-ac54-6d8f8b587624-kube-api-access-mrrb4\" (UniqueName: \"kubernetes.io/projected/8f29a413-6e8a-4a20-ac54-6d8f8b587624-kube-api-access-mrrb4\") pod \"8f29a413-6e8a-4a20-ac54-6d8f8b587624\" (UID: \"8f29a413-6e8a-4a20-ac54-6d8f8b587624\") "
Oct 30 22:10:37.684378 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:37.684352    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f29a413-6e8a-4a20-ac54-6d8f8b587624-kube-api-access-mrrb4" pod "8f29a413-6e8a-4a20-ac54-6d8f8b587624" (UID: "8f29a413-6e8a-4a20-ac54-6d8f8b587624"). InnerVolumeSpecName "kube-api-access-mrrb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:37.787493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:37.785414    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrb4\" (UniqueName: \"kubernetes.io/projected/8f29a413-6e8a-4a20-ac54-6d8f8b587624-kube-api-access-mrrb4\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.252661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.252615    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm" podUID="80767346-e0df-456c-8acf-fe93780e238f" containerName="main" containerID="cri-o://a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f" gracePeriod=45
Oct 30 22:10:38.314781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.314746    4282 generic.go:356] "Generic (PLEG): container finished" podID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" containerID="bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5" exitCode=137
Oct 30 22:10:38.314920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.314791    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:38.314920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.314803    4282 scope.go:122] "RemoveContainer" containerID="bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5"
Oct 30 22:10:38.340831 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:38.340804    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:38.365152 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.365126    4282 scope.go:122] "RemoveContainer" containerID="bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5"
Oct 30 22:10:38.365568 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:38.365494    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5\": container with ID starting with bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5 not found: ID does not exist" containerID="bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5"
Oct 30 22:10:38.365568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.365531    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5"} err="failed to get container status \"bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5\": rpc error: code = NotFound desc = could not find container \"bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5\": container with ID starting with bd1dcc4eb5301e6d17d42ec969496f0eb24277c07b200a17763ff56d075025b5 not found: ID does not exist"
Oct 30 22:10:38.531186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.531164    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:10:38.532915 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:38.532890    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:38.600012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.599978    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-model-cache\") pod \"80767346-e0df-456c-8acf-fe93780e238f\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") "
Oct 30 22:10:38.600616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.600038    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-tmp-dir\") pod \"80767346-e0df-456c-8acf-fe93780e238f\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") "
Oct 30 22:10:38.600616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.600127    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80767346-e0df-456c-8acf-fe93780e238f-tls-certs\" (UniqueName: \"kubernetes.io/secret/80767346-e0df-456c-8acf-fe93780e238f-tls-certs\") pod \"80767346-e0df-456c-8acf-fe93780e238f\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") "
Oct 30 22:10:38.600616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.600254    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-home\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-home\") pod \"80767346-e0df-456c-8acf-fe93780e238f\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") "
Oct 30 22:10:38.600616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.600281    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-model-cache" pod "80767346-e0df-456c-8acf-fe93780e238f" (UID: "80767346-e0df-456c-8acf-fe93780e238f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.600616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.600317    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-dshm\") pod \"80767346-e0df-456c-8acf-fe93780e238f\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") "
Oct 30 22:10:38.600616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.600388    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80767346-e0df-456c-8acf-fe93780e238f-kube-api-access-vtspn\" (UniqueName: \"kubernetes.io/projected/80767346-e0df-456c-8acf-fe93780e238f-kube-api-access-vtspn\") pod \"80767346-e0df-456c-8acf-fe93780e238f\" (UID: \"80767346-e0df-456c-8acf-fe93780e238f\") "
Oct 30 22:10:38.600616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.600565    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-tmp-dir" pod "80767346-e0df-456c-8acf-fe93780e238f" (UID: "80767346-e0df-456c-8acf-fe93780e238f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.600960 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.600736    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-home" pod "80767346-e0df-456c-8acf-fe93780e238f" (UID: "80767346-e0df-456c-8acf-fe93780e238f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.601480 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.601436    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.601616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.601484    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.601616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.601498    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.603299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.603276    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80767346-e0df-456c-8acf-fe93780e238f-kube-api-access-vtspn" pod "80767346-e0df-456c-8acf-fe93780e238f" (UID: "80767346-e0df-456c-8acf-fe93780e238f"). InnerVolumeSpecName "kube-api-access-vtspn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:38.603458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.603429    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80767346-e0df-456c-8acf-fe93780e238f-tls-certs" pod "80767346-e0df-456c-8acf-fe93780e238f" (UID: "80767346-e0df-456c-8acf-fe93780e238f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:38.603458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.603444    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-dshm" pod "80767346-e0df-456c-8acf-fe93780e238f" (UID: "80767346-e0df-456c-8acf-fe93780e238f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.702787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.702754    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtspn\" (UniqueName: \"kubernetes.io/projected/80767346-e0df-456c-8acf-fe93780e238f-kube-api-access-vtspn\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.702787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.702780    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80767346-e0df-456c-8acf-fe93780e238f-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.702787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:38.702790    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80767346-e0df-456c-8acf-fe93780e238f-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:39.093346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.093311    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:10:39.104713 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:39.104667    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:39.116330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.115798    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"]
Oct 30 22:10:39.116330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.116039    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy" containerID="cri-o://17866631305f375d2d8466be634ad51e6d3e33bae02944873bbf0fdbdbc27062" gracePeriod=30
Oct 30 22:10:39.120510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.120051    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"]
Oct 30 22:10:39.120510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.120245    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy" containerID="cri-o://a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392" gracePeriod=30
Oct 30 22:10:39.174914 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:39.174873    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:39.229655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.229617    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"]
Oct 30 22:10:39.229855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.229832    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy" containerID="cri-o://a49bb9e396c60fd0818305841bfe6840372cf2a511680e6fd625316407457d7d" gracePeriod=30
Oct 30 22:10:39.236124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.236094    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"]
Oct 30 22:10:39.236379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.236352    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy" containerID="cri-o://c6813dd14b18bc5a3e3b18891299044a9be596991648f76d324c7dc39ec85a74" gracePeriod=30
Oct 30 22:10:39.260189 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:39.260161    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:39.285274 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:39.285236    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:39.336795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.336760    4282 generic.go:356] "Generic (PLEG): container finished" podID="80767346-e0df-456c-8acf-fe93780e238f" containerID="a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f" exitCode=0
Oct 30 22:10:39.336941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.336804    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm" event={"ID":"80767346-e0df-456c-8acf-fe93780e238f","Type":"ContainerDied","Data":"a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f"}
Oct 30 22:10:39.336941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.336814    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"
Oct 30 22:10:39.336941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.336840    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm" event={"ID":"80767346-e0df-456c-8acf-fe93780e238f","Type":"ContainerDied","Data":"4c7bd2d9002ba55b21e31aeafa83f4242986b74b557dfe175c3d2fdcc028e5c8"}
Oct 30 22:10:39.336941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.336860    4282 scope.go:122] "RemoveContainer" containerID="a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f"
Oct 30 22:10:39.340698 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:39.340667    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:39.372609 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:39.372568    4282 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7691-bbd583\" is forbidden: User \"system:node:ip-10-0-131-245.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-245.us-east-2.compute.internal' and this object" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" pod="opendatahub/test-curl-7691-bbd583"
Oct 30 22:10:39.373645 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.373623    4282 scope.go:122] "RemoveContainer" containerID="a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f"
Oct 30 22:10:39.373990 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:39.373971    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f\": container with ID starting with a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f not found: ID does not exist" containerID="a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f"
Oct 30 22:10:39.374063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.373998    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f"} err="failed to get container status \"a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f\": rpc error: code = NotFound desc = could not find container \"a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f\": container with ID starting with a4162e1313fab83c2d007e1003842ba0f8c2a35eaf7d5c5646f8fcb237b1e22f not found: ID does not exist"
Oct 30 22:10:39.388027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.387514    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"]
Oct 30 22:10:39.392194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.392176    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cca1c7/auth-test-model-cca1c7-kserve-7d99987d5c-hpmlm"]
Oct 30 22:10:39.466426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.466399    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80767346-e0df-456c-8acf-fe93780e238f" path="/var/lib/kubelet/pods/80767346-e0df-456c-8acf-fe93780e238f/volumes"
Oct 30 22:10:39.466861 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:39.466846    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" path="/var/lib/kubelet/pods/8f29a413-6e8a-4a20-ac54-6d8f8b587624/volumes"
Oct 30 22:10:41.389493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.388922    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"]
Oct 30 22:10:41.390115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.390082    4282 cpu_manager.go:414] "containerMap: removing container" podUID="80767346-e0df-456c-8acf-fe93780e238f" containerName="main"
Oct 30 22:10:41.390115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.390109    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80767346-e0df-456c-8acf-fe93780e238f" containerName="main"
Oct 30 22:10:41.390399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.390162    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" containerName="test-curl-7691-bbd583"
Oct 30 22:10:41.390399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.390171    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" containerName="test-curl-7691-bbd583"
Oct 30 22:10:41.390399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.390313    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f29a413-6e8a-4a20-ac54-6d8f8b587624" containerName="test-curl-7691-bbd583"
Oct 30 22:10:41.390399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.390333    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="80767346-e0df-456c-8acf-fe93780e238f" containerName="main"
Oct 30 22:10:41.396224 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.396200    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:41.405031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.404995    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"]
Oct 30 22:10:41.414630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.414078    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"]
Oct 30 22:10:41.414630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.414282    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy" containerID="cri-o://b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf" gracePeriod=30
Oct 30 22:10:41.531491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.489831    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-28b4bb\""
Oct 30 22:10:41.531491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.489967    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-28b4bb-dockercfg-27ngt\""
Oct 30 22:10:41.531491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.492490    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7px22\" (UniqueName: \"kubernetes.io/projected/e94ea251-952e-4a62-bb7d-c39f389542ee-kube-api-access-7px22\") pod \"payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv\" (UID: \"e94ea251-952e-4a62-bb7d-c39f389542ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:41.531491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.492706    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94ea251-952e-4a62-bb7d-c39f389542ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv\" (UID: \"e94ea251-952e-4a62-bb7d-c39f389542ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:41.609493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.609127    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7px22\" (UniqueName: \"kubernetes.io/projected/e94ea251-952e-4a62-bb7d-c39f389542ee-kube-api-access-7px22\") pod \"payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv\" (UID: \"e94ea251-952e-4a62-bb7d-c39f389542ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:41.609493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.609384    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94ea251-952e-4a62-bb7d-c39f389542ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv\" (UID: \"e94ea251-952e-4a62-bb7d-c39f389542ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:41.648551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.646192    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-28b4bb\""
Oct 30 22:10:41.856013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.854081    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7px22\" (UniqueName: \"kubernetes.io/projected/e94ea251-952e-4a62-bb7d-c39f389542ee-kube-api-access-7px22\") pod \"payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv\" (UID: \"e94ea251-952e-4a62-bb7d-c39f389542ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:41.860450 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.860168    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94ea251-952e-4a62-bb7d-c39f389542ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv\" (UID: \"e94ea251-952e-4a62-bb7d-c39f389542ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:41.876418 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:41.876392    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:42.386066 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:42.386035    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"]
Oct 30 22:10:42.386868 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:42.386840    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94ea251_952e_4a62_bb7d_c39f389542ee.slice/crio-54e63c08b06015787558cd9ecb8c5135de7f8d989e0662d79da4cd861eb572c7 WatchSource:0}: Error finding container 54e63c08b06015787558cd9ecb8c5135de7f8d989e0662d79da4cd861eb572c7: Status 404 returned error can't find the container with id 54e63c08b06015787558cd9ecb8c5135de7f8d989e0662d79da4cd861eb572c7
Oct 30 22:10:43.372063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:43.372023    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv" event={"ID":"e94ea251-952e-4a62-bb7d-c39f389542ee","Type":"ContainerStarted","Data":"5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d"}
Oct 30 22:10:43.372628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:43.372086    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:43.372628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:43.372109    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv" event={"ID":"e94ea251-952e-4a62-bb7d-c39f389542ee","Type":"ContainerStarted","Data":"54e63c08b06015787558cd9ecb8c5135de7f8d989e0662d79da4cd861eb572c7"}
Oct 30 22:10:43.393796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:43.393741    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv" podStartSLOduration=2.393727622 podStartE2EDuration="2.393727622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:43.389317611 +0000 UTC m=+1862.047413847" watchObservedRunningTime="2026-10-30 22:10:43.393727622 +0000 UTC m=+1862.051823844"
Oct 30 22:10:43.678343 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:43.678298    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz" podUID="0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" containerName="main" containerID="cri-o://31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651" gracePeriod=45
Oct 30 22:10:44.002425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.002392    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:10:44.108831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.107069    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-dshm\") pod \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") "
Oct 30 22:10:44.108831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.107209    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tls-certs\") pod \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") "
Oct 30 22:10:44.108831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.107304    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-model-cache\") pod \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") "
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.155984    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-kube-api-access-vxdcp\" (UniqueName: \"kubernetes.io/projected/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-kube-api-access-vxdcp\") pod \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") "
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.156085    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-home\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-home\") pod \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") "
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.156142    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tmp-dir\") pod \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\" (UID: \"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9\") "
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.157790    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-model-cache" pod "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" (UID: "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.158355    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-home" pod "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" (UID: "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.158632    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tmp-dir" pod "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" (UID: "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.161772    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-kube-api-access-vxdcp" pod "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" (UID: "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9"). InnerVolumeSpecName "kube-api-access-vxdcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.164118    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-dshm" pod "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" (UID: "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.164458    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.164507    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdcp\" (UniqueName: \"kubernetes.io/projected/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-kube-api-access-vxdcp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.164522    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.164536    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.164544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.164547    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.172531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.169233    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tls-certs" pod "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" (UID: "0702d6a6-7a28-4444-96e3-f53c6c7ac1e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:44.267360 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.266967    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.380528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.378953    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:10:44.393351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.391866    4282 generic.go:356] "Generic (PLEG): container finished" podID="0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" containerID="31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651" exitCode=0
Oct 30 22:10:44.393351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.391908    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"
Oct 30 22:10:44.393351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.392027    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz" event={"ID":"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9","Type":"ContainerDied","Data":"31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651"}
Oct 30 22:10:44.393351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.392057    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz" event={"ID":"0702d6a6-7a28-4444-96e3-f53c6c7ac1e9","Type":"ContainerDied","Data":"e599aec41c09d804048ebbc30bcbb57e5c62fec8a314c17832e48da469ea28d5"}
Oct 30 22:10:44.393351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.392077    4282 scope.go:122] "RemoveContainer" containerID="31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651"
Oct 30 22:10:44.397830 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.397576    4282 generic.go:356] "Generic (PLEG): container finished" podID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerID="a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392" exitCode=0
Oct 30 22:10:44.397830 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.397788    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" event={"ID":"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4","Type":"ContainerDied","Data":"a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392"}
Oct 30 22:10:44.398024 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.397836    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r" event={"ID":"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4","Type":"ContainerDied","Data":"59dd30041019bcbd86628a4444f46abd407d817f5e65f22357648be51588b6bd"}
Oct 30 22:10:44.398024 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.397916    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"
Oct 30 22:10:44.401874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.401109    4282 generic.go:356] "Generic (PLEG): container finished" podID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerID="a49bb9e396c60fd0818305841bfe6840372cf2a511680e6fd625316407457d7d" exitCode=0
Oct 30 22:10:44.401874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.401180    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" event={"ID":"66c2a15e-52fa-4af8-837f-b1a5b0430643","Type":"ContainerDied","Data":"a49bb9e396c60fd0818305841bfe6840372cf2a511680e6fd625316407457d7d"}
Oct 30 22:10:44.406045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.405207    4282 generic.go:356] "Generic (PLEG): container finished" podID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerID="17866631305f375d2d8466be634ad51e6d3e33bae02944873bbf0fdbdbc27062" exitCode=0
Oct 30 22:10:44.406045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.405264    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" event={"ID":"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97","Type":"ContainerDied","Data":"17866631305f375d2d8466be634ad51e6d3e33bae02944873bbf0fdbdbc27062"}
Oct 30 22:10:44.406336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.406317    4282 scope.go:122] "RemoveContainer" containerID="31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651"
Oct 30 22:10:44.406776 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:44.406750    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651\": container with ID starting with 31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651 not found: ID does not exist" containerID="31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651"
Oct 30 22:10:44.406826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.406793    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651"} err="failed to get container status \"31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651\": rpc error: code = NotFound desc = could not find container \"31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651\": container with ID starting with 31519b2b6c9d9c38cbf335a9d8cb673652f53c57287178a03c6fdb69f0e37651 not found: ID does not exist"
Oct 30 22:10:44.406826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.406818    4282 scope.go:122] "RemoveContainer" containerID="a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392"
Oct 30 22:10:44.408271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.408246    4282 generic.go:356] "Generic (PLEG): container finished" podID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerID="c6813dd14b18bc5a3e3b18891299044a9be596991648f76d324c7dc39ec85a74" exitCode=0
Oct 30 22:10:44.409545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.409437    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" event={"ID":"a1660a82-f833-45b3-883c-9a34c1ec42a6","Type":"ContainerDied","Data":"c6813dd14b18bc5a3e3b18891299044a9be596991648f76d324c7dc39ec85a74"}
Oct 30 22:10:44.429644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.427352    4282 scope.go:122] "RemoveContainer" containerID="a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392"
Oct 30 22:10:44.429644 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:44.427857    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392\": container with ID starting with a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392 not found: ID does not exist" containerID="a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392"
Oct 30 22:10:44.429644 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.427893    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392"} err="failed to get container status \"a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392\": rpc error: code = NotFound desc = could not find container \"a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392\": container with ID starting with a6c484f426a13ad8b794868e1301e803bd787ba45f3a768e1761a4d36f7f6392 not found: ID does not exist"
Oct 30 22:10:44.446096 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.446071    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"]
Oct 30 22:10:44.455775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.455712    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0f73c9/auth-test-model-0f73c9-kserve-6f9fb5596f-7wghz"]
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.469984    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-podinfo\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.470111    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-envoy\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.470245    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-token\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-token\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.470339    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-certs\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.470384    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istiod-ca-cert\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.470489    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-kube-api-access-mc78p\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-kube-api-access-mc78p\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.470608    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-credential-socket\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.470645    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-socket\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.471505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.470714    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-data\") pod \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\" (UID: \"75bd9566-2ab1-4aeb-b273-f6ad727bd1d4\") "
Oct 30 22:10:44.472101 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.472026    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-certs" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.473131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.472416    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-data" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.473131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.472678    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-socket" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.473131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.472743    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-credential-socket" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.473131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.472919    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istiod-ca-cert" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:44.476013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.475956    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-envoy" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.476188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.476156    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-podinfo" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:44.476334 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.476316    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-token" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.476884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.476855    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-kube-api-access-mc78p" pod "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" (UID: "75bd9566-2ab1-4aeb-b273-f6ad727bd1d4"). InnerVolumeSpecName "kube-api-access-mc78p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.562173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.561813    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:10:44.575524 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575502    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.575648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575527    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.575648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575545    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.575648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575558    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.575648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575572    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc78p\" (UniqueName: \"kubernetes.io/projected/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-kube-api-access-mc78p\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.575648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575587    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.575648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575600    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.575648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575613    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.575648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.575625    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.638117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.638093    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 22:10:44.676723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.676691    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-podinfo\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.676847 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.676796    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c2a15e-52fa-4af8-837f-b1a5b0430643-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c2a15e-52fa-4af8-837f-b1a5b0430643-istiod-ca-cert\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.676956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.676939    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-kube-api-access-7rjfz\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-kube-api-access-7rjfz\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.677008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.676980    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-certs\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.677065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677025    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-socket\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.677220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677202    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-envoy\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.677288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677236    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-token\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-token\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.677288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677266    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-data\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.677398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677309    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c2a15e-52fa-4af8-837f-b1a5b0430643-istiod-ca-cert" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:44.677398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677336    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-credential-socket\") pod \"66c2a15e-52fa-4af8-837f-b1a5b0430643\" (UID: \"66c2a15e-52fa-4af8-837f-b1a5b0430643\") "
Oct 30 22:10:44.677591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677572    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-socket" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.677940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677748    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-certs" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.677940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.677918    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-data" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.678125 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.678065    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-credential-socket" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.679113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.679090    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.679267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.679252    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.679366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.679354    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c2a15e-52fa-4af8-837f-b1a5b0430643-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.679541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.679527    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.679647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.679634    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.681084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.680584    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-podinfo" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:44.681084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.680740    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-envoy" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.682055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.682014    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-kube-api-access-7rjfz" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "kube-api-access-7rjfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.682778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.682753    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-token" pod "66c2a15e-52fa-4af8-837f-b1a5b0430643" (UID: "66c2a15e-52fa-4af8-837f-b1a5b0430643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.746355 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.746327    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"]
Oct 30 22:10:44.753700 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.753671    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-psc5r"]
Oct 30 22:10:44.781431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.781397    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-certs\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.781680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.781654    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-token\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-token\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.781771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.781719    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-credential-socket\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.781771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.781741    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-certs" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.781951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.781801    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-data\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.781951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.781863    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1660a82-f833-45b3-883c-9a34c1ec42a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1660a82-f833-45b3-883c-9a34c1ec42a6-istiod-ca-cert\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.781951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.781905    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-envoy\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.782129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.781951    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-podinfo\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.782188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.782114    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-kube-api-access-rh574\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-kube-api-access-rh574\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.782497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.782441    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1660a82-f833-45b3-883c-9a34c1ec42a6-istiod-ca-cert" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:44.782594 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.782525    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-socket\") pod \"a1660a82-f833-45b3-883c-9a34c1ec42a6\" (UID: \"a1660a82-f833-45b3-883c-9a34c1ec42a6\") "
Oct 30 22:10:44.782772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.782753    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-credential-socket" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.782965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.782940    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-data" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.783678 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.783645    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.783833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.783817    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.783983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.783967    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.784111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.784097    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rjfz\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-kube-api-access-7rjfz\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.784205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.784194    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.784307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.784292    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1660a82-f833-45b3-883c-9a34c1ec42a6-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.784425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.784412    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.784539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.784526    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c2a15e-52fa-4af8-837f-b1a5b0430643-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.784643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.783866    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-socket" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.786629 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.786557    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-podinfo" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:44.789767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.789707    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-kube-api-access-rh574" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "kube-api-access-rh574". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.789767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.789738    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-token" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:44.791745 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.791719    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-envoy" pod "a1660a82-f833-45b3-883c-9a34c1ec42a6" (UID: "a1660a82-f833-45b3-883c-9a34c1ec42a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:44.895416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.894774    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.895416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.894814    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.895416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.894830    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1660a82-f833-45b3-883c-9a34c1ec42a6-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.895416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.894844    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh574\" (UniqueName: \"kubernetes.io/projected/a1660a82-f833-45b3-883c-9a34c1ec42a6-kube-api-access-rh574\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.895416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.894860    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1660a82-f833-45b3-883c-9a34c1ec42a6-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:44.958054 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.958019    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 22:10:44.999963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.999869    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-certs\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:44.999963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:44.999936    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-credential-socket\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:45.000197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.000113    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-data\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:45.000197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.000149    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-token\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-token\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:45.000298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.000195    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-socket\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:45.000298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.000237    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-kube-api-access-nl2sg\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-kube-api-access-nl2sg\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:45.000298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.000272    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-podinfo\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:45.000433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.000352    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istiod-ca-cert\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:45.000531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.000512    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-envoy\") pod \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\" (UID: \"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97\") "
Oct 30 22:10:45.001101 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.001068    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-socket" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:45.001287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.001260    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-data" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:45.001378 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.001279    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-credential-socket" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:45.001378 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.001296    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istiod-ca-cert" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:45.001582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.001551    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-certs" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:45.002094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.002068    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.002460 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.002100    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.002460 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.002122    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.002460 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.002135    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.002460 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.002155    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.014425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.011844    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-token" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:45.014425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.011887    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-kube-api-access-nl2sg" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "kube-api-access-nl2sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:45.014425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.011977    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-podinfo" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:45.014425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.013540    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-envoy" pod "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" (UID: "45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:45.104679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.104643    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.104679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.104683    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.104880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.104698    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.104880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.104713    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl2sg\" (UniqueName: \"kubernetes.io/projected/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97-kube-api-access-nl2sg\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:45.421542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.421502    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz" event={"ID":"66c2a15e-52fa-4af8-837f-b1a5b0430643","Type":"ContainerDied","Data":"caadabe7b241a8993f2f1504296902508260eca6ef9ee23b01e8e9ecc33eff91"}
Oct 30 22:10:45.422053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.421565    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"
Oct 30 22:10:45.422053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.421613    4282 scope.go:122] "RemoveContainer" containerID="a49bb9e396c60fd0818305841bfe6840372cf2a511680e6fd625316407457d7d"
Oct 30 22:10:45.423182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.423155    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9" event={"ID":"45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97","Type":"ContainerDied","Data":"ec451ae249e328b29a37c5cc29651b500aec286ee7ef45d32b7cf58a5ecac385"}
Oct 30 22:10:45.423332 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.423163    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"
Oct 30 22:10:45.425549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.425461    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc" event={"ID":"a1660a82-f833-45b3-883c-9a34c1ec42a6","Type":"ContainerDied","Data":"cf7a2bfbe3797df65f37d8f4fd276971c8e233c807bfd7ea5bbbd90b639da433"}
Oct 30 22:10:45.425549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.425504    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"
Oct 30 22:10:45.442636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.442619    4282 scope.go:122] "RemoveContainer" containerID="17866631305f375d2d8466be634ad51e6d3e33bae02944873bbf0fdbdbc27062"
Oct 30 22:10:45.454712 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.454693    4282 scope.go:122] "RemoveContainer" containerID="c6813dd14b18bc5a3e3b18891299044a9be596991648f76d324c7dc39ec85a74"
Oct 30 22:10:45.456218 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.456197    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"]
Oct 30 22:10:45.468632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.468608    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" path="/var/lib/kubelet/pods/0702d6a6-7a28-4444-96e3-f53c6c7ac1e9/volumes"
Oct 30 22:10:45.469181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.469167    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" path="/var/lib/kubelet/pods/75bd9566-2ab1-4aeb-b273-f6ad727bd1d4/volumes"
Oct 30 22:10:45.469536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.469520    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lcmfz"]
Oct 30 22:10:45.475250 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.475231    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"]
Oct 30 22:10:45.481229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.481211    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-2wxqc"]
Oct 30 22:10:45.490506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.490488    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"]
Oct 30 22:10:45.497008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:45.496990    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-xnlm9"]
Oct 30 22:10:47.175970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.175942    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 22:10:47.273765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.273690    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-credential-socket\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.273765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.273733    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-certs\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.273996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.273769    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-socket\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.273996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.273836    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-token\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-token\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.273996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.273877    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6e9352-4eac-4838-9640-612076e9b0e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6e9352-4eac-4838-9640-612076e9b0e6-istiod-ca-cert\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.273996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.273931    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-podinfo\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.274196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.274096    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-data\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.274196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.274144    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-socket" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:47.274196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.274185    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-kube-api-access-kb9mr\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-kube-api-access-kb9mr\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.274369 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.274230    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-envoy\") pod \"8e6e9352-4eac-4838-9640-612076e9b0e6\" (UID: \"8e6e9352-4eac-4838-9640-612076e9b0e6\") "
Oct 30 22:10:47.274369 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.274340    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-certs" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:47.275072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.274644    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-credential-socket" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:47.275072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.274836    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-data" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:47.275276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.275146    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.275276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.275169    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.275276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.275185    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.275276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.275199    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.275901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.275866    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6e9352-4eac-4838-9640-612076e9b0e6-istiod-ca-cert" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:47.277209 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.277180    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-token" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:47.283519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.283116    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-podinfo" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:47.283519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.283409    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-envoy" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:47.284220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.283589    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-kube-api-access-kb9mr" pod "8e6e9352-4eac-4838-9640-612076e9b0e6" (UID: "8e6e9352-4eac-4838-9640-612076e9b0e6"). InnerVolumeSpecName "kube-api-access-kb9mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:47.377051 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.377021    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6e9352-4eac-4838-9640-612076e9b0e6-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.377051 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.377053    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.377258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.377071    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb9mr\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-kube-api-access-kb9mr\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.377258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.377087    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.377258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.377114    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6e9352-4eac-4838-9640-612076e9b0e6-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:47.449286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.449255    4282 generic.go:356] "Generic (PLEG): container finished" podID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerID="b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf" exitCode=0
Oct 30 22:10:47.449489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.449307    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"
Oct 30 22:10:47.449489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.449323    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" event={"ID":"8e6e9352-4eac-4838-9640-612076e9b0e6","Type":"ContainerDied","Data":"b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf"}
Oct 30 22:10:47.449489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.449378    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f" event={"ID":"8e6e9352-4eac-4838-9640-612076e9b0e6","Type":"ContainerDied","Data":"d5c54a4179c19ef9095463e85411772175d4cd154356ab9b2c487e2ec0e1908b"}
Oct 30 22:10:47.449489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.449409    4282 scope.go:122] "RemoveContainer" containerID="b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf"
Oct 30 22:10:47.462215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.462196    4282 scope.go:122] "RemoveContainer" containerID="b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf"
Oct 30 22:10:47.462512 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:47.462484    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf\": container with ID starting with b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf not found: ID does not exist" containerID="b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf"
Oct 30 22:10:47.462577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.462525    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf"} err="failed to get container status \"b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf\": rpc error: code = NotFound desc = could not find container \"b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf\": container with ID starting with b1d0212bfb9946f5932d9f63875d39d75280861f34bc21e4368c230933fb9edf not found: ID does not exist"
Oct 30 22:10:47.467621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.467602    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" path="/var/lib/kubelet/pods/45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97/volumes"
Oct 30 22:10:47.468076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.468063    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" path="/var/lib/kubelet/pods/66c2a15e-52fa-4af8-837f-b1a5b0430643/volumes"
Oct 30 22:10:47.468577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.468562    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" path="/var/lib/kubelet/pods/a1660a82-f833-45b3-883c-9a34c1ec42a6/volumes"
Oct 30 22:10:47.485602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.485574    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"]
Oct 30 22:10:47.493598 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:47.493569    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zpq8f"]
Oct 30 22:10:48.286067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286033    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"]
Oct 30 22:10:48.286768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286753    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy"
Oct 30 22:10:48.286818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286769    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy"
Oct 30 22:10:48.286818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286789    4282 cpu_manager.go:414] "containerMap: removing container" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy"
Oct 30 22:10:48.286818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286794    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy"
Oct 30 22:10:48.286818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286818    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy"
Oct 30 22:10:48.286950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286823    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy"
Oct 30 22:10:48.286950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286835    4282 cpu_manager.go:414] "containerMap: removing container" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy"
Oct 30 22:10:48.286950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286839    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy"
Oct 30 22:10:48.286950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286846    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" containerName="main"
Oct 30 22:10:48.286950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286850    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" containerName="main"
Oct 30 22:10:48.286950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286869    4282 cpu_manager.go:414] "containerMap: removing container" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy"
Oct 30 22:10:48.286950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286873    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy"
Oct 30 22:10:48.287144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286965    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c2a15e-52fa-4af8-837f-b1a5b0430643" containerName="istio-proxy"
Oct 30 22:10:48.287144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286978    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" containerName="istio-proxy"
Oct 30 22:10:48.287144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286985    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a17208-d1ac-4c41-b5f4-8a4e3bbd3a97" containerName="istio-proxy"
Oct 30 22:10:48.287144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.286992    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0702d6a6-7a28-4444-96e3-f53c6c7ac1e9" containerName="main"
Oct 30 22:10:48.287144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.287001    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="75bd9566-2ab1-4aeb-b273-f6ad727bd1d4" containerName="istio-proxy"
Oct 30 22:10:48.287144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.287008    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1660a82-f833-45b3-883c-9a34c1ec42a6" containerName="istio-proxy"
Oct 30 22:10:48.292238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.292219    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.296424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.296403    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-9c8ecf-openshift-default-dockercfg-6qtv6\""
Oct 30 22:10:48.304074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.303417    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"]
Oct 30 22:10:48.398199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398159    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-podinfo\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.398199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398209    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-envoy\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.398401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398304    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.398401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398325    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.398401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398357    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-certs\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.398528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398461    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-token\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.398528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398520    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-credential-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.398597 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398575    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-data\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.398637 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.398630    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wrq\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-kube-api-access-t8wrq\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.499907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.499880    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-podinfo\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.499917    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-envoy\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.499957    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.499974    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500001    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-certs\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500084    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-token\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500127    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-credential-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500215    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-data\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500266    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wrq\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-kube-api-access-t8wrq\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500407    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500414    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-certs\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500709    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-data\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.500771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.500726    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-credential-socket\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.501314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.501288    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istiod-ca-cert\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.502733 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.502711    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-podinfo\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.502940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.502923    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-envoy\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.511171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.511148    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-token\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.511653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.511632    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wrq\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-kube-api-access-t8wrq\") pod \"e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") " pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:48.610012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:48.609929    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:49.001586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.001559    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"]
Oct 30 22:10:49.004022 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:49.003995    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd7a3fd_f269_4d19_a42a_0899b8e8104c.slice/crio-c7c3ec58290f6f9191fdc9af2b9c6d6a9134e624704b26246227646499cd3008 WatchSource:0}: Error finding container c7c3ec58290f6f9191fdc9af2b9c6d6a9134e624704b26246227646499cd3008: Status 404 returned error can't find the container with id c7c3ec58290f6f9191fdc9af2b9c6d6a9134e624704b26246227646499cd3008
Oct 30 22:10:49.006967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.006932    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:49.007070 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.007000    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:49.007070 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.007029    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:10:49.513167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.513112    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6e9352-4eac-4838-9640-612076e9b0e6" path="/var/lib/kubelet/pods/8e6e9352-4eac-4838-9640-612076e9b0e6/volumes"
Oct 30 22:10:49.514232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.514209    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" event={"ID":"6cd7a3fd-f269-4d19-a42a-0899b8e8104c","Type":"ContainerStarted","Data":"f1003e352c6b1a734a3321474142f9eab9936c01c196b367f3c1bf601d7bff93"}
Oct 30 22:10:49.514385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.514370    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" event={"ID":"6cd7a3fd-f269-4d19-a42a-0899b8e8104c","Type":"ContainerStarted","Data":"c7c3ec58290f6f9191fdc9af2b9c6d6a9134e624704b26246227646499cd3008"}
Oct 30 22:10:49.548190 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.548140    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" podStartSLOduration=1.54812558 podStartE2EDuration="1.54812558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:49.538837163 +0000 UTC m=+1868.196933392" watchObservedRunningTime="2026-10-30 22:10:49.54812558 +0000 UTC m=+1868.206221803"
Oct 30 22:10:49.630511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:49.619593    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:50.203613 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.203580    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"]
Oct 30 22:10:50.209890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.209867    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.221633 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.216958    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-b9d868-serving-cert\""
Oct 30 22:10:50.227225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.227201    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"]
Oct 30 22:10:50.246567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.246541    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.246702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.246586    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.246787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.246726    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-metrics-certs\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.246914 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.246807    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469jd\" (UniqueName: \"kubernetes.io/projected/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-kube-api-access-469jd\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.383505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.381792    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.383505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.381855    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.383505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.381999    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-metrics-certs\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.383505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.382089    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469jd\" (UniqueName: \"kubernetes.io/projected/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-kube-api-access-469jd\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.383966 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:50.383619    4282 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-serving-cert: secret "maas-api-e2e-extmod-a-b9d868-serving-cert" not found
Oct 30 22:10:50.383966 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:10:50.383693    4282 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls podName:84b39bf1-07b7-46c4-bcf7-3b955a955bd8 nodeName:}" failed. No retries permitted until 2026-10-30 22:10:50.883671561 +0000 UTC m=+1869.541767775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls") pod "maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr" (UID: "84b39bf1-07b7-46c4-bcf7-3b955a955bd8") : secret "maas-api-e2e-extmod-a-b9d868-serving-cert" not found
Oct 30 22:10:50.387643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.384615    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.433244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.421246    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-metrics-certs\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.435527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.435410    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-469jd\" (UniqueName: \"kubernetes.io/projected/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-kube-api-access-469jd\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.612268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.612171    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:10:50.612757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.612252    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:10:50.914506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.900563    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:50.966161 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:50.966124    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:51.138500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:51.138235    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:51.527089 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:51.527053    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"]
Oct 30 22:10:51.533587 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:51.533559    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b39bf1_07b7_46c4_bcf7_3b955a955bd8.slice/crio-445e373f6e6fa6e1b02e1635addc0e6403f25641acff3446caa3e26a4093d8b9 WatchSource:0}: Error finding container 445e373f6e6fa6e1b02e1635addc0e6403f25641acff3446caa3e26a4093d8b9: Status 404 returned error can't find the container with id 445e373f6e6fa6e1b02e1635addc0e6403f25641acff3446caa3e26a4093d8b9
Oct 30 22:10:51.535806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:51.535785    4282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:10:51.611145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:51.611113    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:10:51.611295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:51.611169    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:10:52.551740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:52.551700    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr" event={"ID":"84b39bf1-07b7-46c4-bcf7-3b955a955bd8","Type":"ContainerStarted","Data":"f4bedda2f34f0923475422c16af6fd1f61aa1165651b87050695a12889d67eeb"}
Oct 30 22:10:52.552365 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:52.552339    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr" event={"ID":"84b39bf1-07b7-46c4-bcf7-3b955a955bd8","Type":"ContainerStarted","Data":"445e373f6e6fa6e1b02e1635addc0e6403f25641acff3446caa3e26a4093d8b9"}
Oct 30 22:10:52.560732 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:52.560710    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:10:52.588130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:52.588076    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr" podStartSLOduration=2.083701003 podStartE2EDuration="2.588063017s" totalImagesPullingTime="504.362014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:52.585402513 +0000 UTC m=+1871.243498735" watchObservedRunningTime="2026-10-30 22:10:52.588063017 +0000 UTC m=+1871.246159240"
Oct 30 22:10:52.610902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:52.610878    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:10:52.611573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:52.611543    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:10:53.610966 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.610930    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:10:53.611358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.610979    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:10:53.722748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.722115    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"]
Oct 30 22:10:53.729633 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.728853    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.740848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.740820    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-28b4bb-serving-cert\""
Oct 30 22:10:53.741280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.741260    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"]
Oct 30 22:10:53.807518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.807485    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4x9h\" (UniqueName: \"kubernetes.io/projected/5b7ec276-992d-40e4-a83e-65b55a7f7df7-kube-api-access-z4x9h\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.807753 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.807539    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-metrics-certs\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.807753 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.807692    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.807906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.807863    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7ec276-992d-40e4-a83e-65b55a7f7df7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.963495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.922038    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.998494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.996214    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7ec276-992d-40e4-a83e-65b55a7f7df7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.998494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.996383    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4x9h\" (UniqueName: \"kubernetes.io/projected/5b7ec276-992d-40e4-a83e-65b55a7f7df7-kube-api-access-z4x9h\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.998494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.996442    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-metrics-certs\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:53.998494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:53.998308    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7ec276-992d-40e4-a83e-65b55a7f7df7-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:54.018187 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.018149    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-maas-api-tls\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:54.023028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.020092    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-metrics-certs\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:54.023028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.021744    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4x9h\" (UniqueName: \"kubernetes.io/projected/5b7ec276-992d-40e4-a83e-65b55a7f7df7-kube-api-access-z4x9h\") pod \"maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:54.086389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.086353    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:54.395180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.395150    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"]
Oct 30 22:10:54.396156 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:10:54.396133    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7ec276_992d_40e4_a83e_65b55a7f7df7.slice/crio-8d22d195ce8e03c0f4b12daa2d57299cbc104f3dc3268a4796d0c58e52a6b480 WatchSource:0}: Error finding container 8d22d195ce8e03c0f4b12daa2d57299cbc104f3dc3268a4796d0c58e52a6b480: Status 404 returned error can't find the container with id 8d22d195ce8e03c0f4b12daa2d57299cbc104f3dc3268a4796d0c58e52a6b480
Oct 30 22:10:54.413409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.413386    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:10:54.628493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.610616    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:10:54.629143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.629105    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:10:54.681528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.677165    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"]
Oct 30 22:10:54.699681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.697411    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy" containerID="cri-o://9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4" gracePeriod=30
Oct 30 22:10:54.905758 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:54.905302    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz" event={"ID":"5b7ec276-992d-40e4-a83e-65b55a7f7df7","Type":"ContainerStarted","Data":"8d22d195ce8e03c0f4b12daa2d57299cbc104f3dc3268a4796d0c58e52a6b480"}
Oct 30 22:10:55.610641 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:55.610603    4282 patch_prober.go:29] interesting pod/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 22:10:55.610837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:55.610672    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 22:10:55.914056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:55.913960    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz" event={"ID":"5b7ec276-992d-40e4-a83e-65b55a7f7df7","Type":"ContainerStarted","Data":"49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d"}
Oct 30 22:10:55.914056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:55.914053    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:10:55.945049 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:55.944996    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz" podStartSLOduration=2.4019551679999998 podStartE2EDuration="2.944982818s" totalImagesPullingTime="543.02765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:55.940153497 +0000 UTC m=+1874.598249719" watchObservedRunningTime="2026-10-30 22:10:55.944982818 +0000 UTC m=+1874.603079047"
Oct 30 22:10:56.310631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:56.310592    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"]
Oct 30 22:10:56.620343 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:56.620253    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:56.620661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:56.620633    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:56.621522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:56.621502    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:10:59.711607 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.711570    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"]
Oct 30 22:10:59.718004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.717978    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.724038 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.724011    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-f0f5e2-openshift-default-dockercfg-q28pr\""
Oct 30 22:10:59.747278 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.747249    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"]
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.884578    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-envoy\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.884667    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-credential-socket\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.884757    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-token\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.884952    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-podinfo\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.885002    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-socket\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.885045    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-kube-api-access-6gcpp\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.885087    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-certs\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.885125    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istiod-ca-cert\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.885514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.885206    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-data\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:10:59.992630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:10:59.988581    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007384    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-podinfo\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007456    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-socket\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007526    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-kube-api-access-6gcpp\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007574    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-certs\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007613    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istiod-ca-cert\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007696    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-data\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007805    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-envoy\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007855    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-credential-socket\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.007943    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-token\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.009109    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-data\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.010348    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istiod-ca-cert\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.010780    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-socket\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.011354    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-certs\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.015823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.011725    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-credential-socket\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.061757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.061135    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-token\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.085550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.070860    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-envoy\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.085550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.071627    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-podinfo\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.128238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.097149    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-kube-api-access-6gcpp\") pod \"e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") " pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.354493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.351808    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:00.477569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.476410    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:11:00.574874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.574793    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-credential-socket\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.574874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.574835    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-certs\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.574874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.574859    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f55e61-ab06-4d34-97e7-2d94331f99ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f55e61-ab06-4d34-97e7-2d94331f99ce-istiod-ca-cert\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.575181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575024    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-envoy\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.575181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575124    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-token\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-token\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.575300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575095    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-credential-socket" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.575300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575157    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-certs" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.575414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575335    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-kube-api-access-jz88d\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-kube-api-access-jz88d\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.575414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575369    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f55e61-ab06-4d34-97e7-2d94331f99ce-istiod-ca-cert" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:00.575582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575435    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-socket\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.575582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575457    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-podinfo\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.575582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575522    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-data\") pod \"73f55e61-ab06-4d34-97e7-2d94331f99ce\" (UID: \"73f55e61-ab06-4d34-97e7-2d94331f99ce\") "
Oct 30 22:11:00.575749 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575696    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-socket" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.576964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.575949    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-data" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.576964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.576313    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.576964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.576328    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.576964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.576336    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.576964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.576346    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.576964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.576373    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73f55e61-ab06-4d34-97e7-2d94331f99ce-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.577947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.577926    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-envoy" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:00.578187 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.578158    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-podinfo" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:00.578258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.578163    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-kube-api-access-jz88d" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "kube-api-access-jz88d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:00.578258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.578245    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-token" pod "73f55e61-ab06-4d34-97e7-2d94331f99ce" (UID: "73f55e61-ab06-4d34-97e7-2d94331f99ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:00.748417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.748381    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.749204 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.749183    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.749449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.749431    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz88d\" (UniqueName: \"kubernetes.io/projected/73f55e61-ab06-4d34-97e7-2d94331f99ce-kube-api-access-jz88d\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.749597 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.749582    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73f55e61-ab06-4d34-97e7-2d94331f99ce-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:00.984493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:00.984204    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"]
Oct 30 22:11:01.009585 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:11:01.009552    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf021a87d_4a02_4e67_b8c5_1c23d125d78a.slice/crio-a589b0d9e88c575029e885c7481975afbe3863a4a174ecab4c3a79c0aec51c25 WatchSource:0}: Error finding container a589b0d9e88c575029e885c7481975afbe3863a4a174ecab4c3a79c0aec51c25: Status 404 returned error can't find the container with id a589b0d9e88c575029e885c7481975afbe3863a4a174ecab4c3a79c0aec51c25
Oct 30 22:11:01.013426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.013388    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:01.013565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.013516    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:01.013565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.013556    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:01.174628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.174589    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" event={"ID":"f021a87d-4a02-4e67-b8c5-1c23d125d78a","Type":"ContainerStarted","Data":"24cf27d0c576a34112da182a097e34d04e21d6885d1492457d223854b848b968"}
Oct 30 22:11:01.174842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.174653    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" event={"ID":"f021a87d-4a02-4e67-b8c5-1c23d125d78a","Type":"ContainerStarted","Data":"a589b0d9e88c575029e885c7481975afbe3863a4a174ecab4c3a79c0aec51c25"}
Oct 30 22:11:01.176755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.176539    4282 generic.go:356] "Generic (PLEG): container finished" podID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerID="9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4" exitCode=0
Oct 30 22:11:01.176755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.176601    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"
Oct 30 22:11:01.176755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.176717    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" event={"ID":"73f55e61-ab06-4d34-97e7-2d94331f99ce","Type":"ContainerDied","Data":"9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4"}
Oct 30 22:11:01.177008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.176768    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz" event={"ID":"73f55e61-ab06-4d34-97e7-2d94331f99ce","Type":"ContainerDied","Data":"70e47d032555db36cd6c017a15bd8151b47114a25831e75ffb051421343665c7"}
Oct 30 22:11:01.177008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.176792    4282 scope.go:122] "RemoveContainer" containerID="9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4"
Oct 30 22:11:01.189259 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.188980    4282 scope.go:122] "RemoveContainer" containerID="9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4"
Oct 30 22:11:01.189336 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:11:01.189277    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4\": container with ID starting with 9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4 not found: ID does not exist" containerID="9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4"
Oct 30 22:11:01.189336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.189300    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4"} err="failed to get container status \"9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4\": rpc error: code = NotFound desc = could not find container \"9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4\": container with ID starting with 9061f9ecbab97f55e0e0d6456e6955fc749dba565a21b034c929135d876323a4 not found: ID does not exist"
Oct 30 22:11:01.202695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.202637    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podStartSLOduration=2.202618995 podStartE2EDuration="2.202618995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:01.200936879 +0000 UTC m=+1879.859033102" watchObservedRunningTime="2026-10-30 22:11:01.202618995 +0000 UTC m=+1879.860715218"
Oct 30 22:11:01.225254 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.225223    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"]
Oct 30 22:11:01.230769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.230744    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-77xxz"]
Oct 30 22:11:01.358006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.357908    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:01.467054 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.467011    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" path="/var/lib/kubelet/pods/73f55e61-ab06-4d34-97e7-2d94331f99ce/volumes"
Oct 30 22:11:01.915210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.915181    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"]
Oct 30 22:11:01.915937 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.915922    4282 cpu_manager.go:414] "containerMap: removing container" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy"
Oct 30 22:11:01.915979 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.915938    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy"
Oct 30 22:11:01.916078 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.916069    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f55e61-ab06-4d34-97e7-2d94331f99ce" containerName="istio-proxy"
Oct 30 22:11:01.921344 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.921292    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:01.936268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.936230    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a1ea1e-dockercfg-wxrvv\""
Oct 30 22:11:01.936268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.936241    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a1ea1e\""
Oct 30 22:11:01.936791 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.936777    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"]
Oct 30 22:11:01.941843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.941813    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"]
Oct 30 22:11:01.942033 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.942012    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:01.965714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:01.965684    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"]
Oct 30 22:11:02.050702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.050676    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:11:02.272046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.272003    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpcjr\" (UniqueName: \"kubernetes.io/projected/0438efc5-012a-4fb4-98cf-7b8b9af86145-kube-api-access-rpcjr\") pod \"payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t\" (UID: \"0438efc5-012a-4fb4-98cf-7b8b9af86145\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:02.290493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.272377    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5fg\" (UniqueName: \"kubernetes.io/projected/2486a5ab-245c-40cb-93d0-c01ab6408012-kube-api-access-pv5fg\") pod \"payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55\" (UID: \"2486a5ab-245c-40cb-93d0-c01ab6408012\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:02.290891 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.290853    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2486a5ab-245c-40cb-93d0-c01ab6408012-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55\" (UID: \"2486a5ab-245c-40cb-93d0-c01ab6408012\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:02.463954 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.463914    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438efc5-012a-4fb4-98cf-7b8b9af86145-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t\" (UID: \"0438efc5-012a-4fb4-98cf-7b8b9af86145\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:02.525226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.516974    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:02.565677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.565647    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:02.566300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.566271    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpcjr\" (UniqueName: \"kubernetes.io/projected/0438efc5-012a-4fb4-98cf-7b8b9af86145-kube-api-access-rpcjr\") pod \"payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t\" (UID: \"0438efc5-012a-4fb4-98cf-7b8b9af86145\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:02.566433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.566364    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5fg\" (UniqueName: \"kubernetes.io/projected/2486a5ab-245c-40cb-93d0-c01ab6408012-kube-api-access-pv5fg\") pod \"payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55\" (UID: \"2486a5ab-245c-40cb-93d0-c01ab6408012\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:02.566567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.566436    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2486a5ab-245c-40cb-93d0-c01ab6408012-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55\" (UID: \"2486a5ab-245c-40cb-93d0-c01ab6408012\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:02.566646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.566585    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438efc5-012a-4fb4-98cf-7b8b9af86145-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t\" (UID: \"0438efc5-012a-4fb4-98cf-7b8b9af86145\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:02.567669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.567646    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2486a5ab-245c-40cb-93d0-c01ab6408012-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55\" (UID: \"2486a5ab-245c-40cb-93d0-c01ab6408012\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:02.568341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.568316    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438efc5-012a-4fb4-98cf-7b8b9af86145-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t\" (UID: \"0438efc5-012a-4fb4-98cf-7b8b9af86145\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:02.568795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.568774    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:02.579217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.579190    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpcjr\" (UniqueName: \"kubernetes.io/projected/0438efc5-012a-4fb4-98cf-7b8b9af86145-kube-api-access-rpcjr\") pod \"payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t\" (UID: \"0438efc5-012a-4fb4-98cf-7b8b9af86145\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:02.579617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.579599    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5fg\" (UniqueName: \"kubernetes.io/projected/2486a5ab-245c-40cb-93d0-c01ab6408012-kube-api-access-pv5fg\") pod \"payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55\" (UID: \"2486a5ab-245c-40cb-93d0-c01ab6408012\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:02.842229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.842141    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:02.944420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:02.934419    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:03.254643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.254613    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"]
Oct 30 22:11:03.255251 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:11:03.255035    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2486a5ab_245c_40cb_93d0_c01ab6408012.slice/crio-7d505fb789f8e39dd0e0173007cbdb00ec6ad47b3226012f488f4042b482a0ad WatchSource:0}: Error finding container 7d505fb789f8e39dd0e0173007cbdb00ec6ad47b3226012f488f4042b482a0ad: Status 404 returned error can't find the container with id 7d505fb789f8e39dd0e0173007cbdb00ec6ad47b3226012f488f4042b482a0ad
Oct 30 22:11:03.304243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.304202    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"]
Oct 30 22:11:03.656550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.647569    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t" event={"ID":"0438efc5-012a-4fb4-98cf-7b8b9af86145","Type":"ContainerStarted","Data":"0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07"}
Oct 30 22:11:03.656550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.647635    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t" event={"ID":"0438efc5-012a-4fb4-98cf-7b8b9af86145","Type":"ContainerStarted","Data":"fc8ffdba4c0ef270fa7e5af75c72052e3bf8f7949794e623f384252b2b076824"}
Oct 30 22:11:03.656550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.648917    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:03.792487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.785818    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55" event={"ID":"2486a5ab-245c-40cb-93d0-c01ab6408012","Type":"ContainerStarted","Data":"756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d"}
Oct 30 22:11:03.792487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.785884    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55" event={"ID":"2486a5ab-245c-40cb-93d0-c01ab6408012","Type":"ContainerStarted","Data":"7d505fb789f8e39dd0e0173007cbdb00ec6ad47b3226012f488f4042b482a0ad"}
Oct 30 22:11:03.792487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.785930    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:03.792487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.790217    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t" podStartSLOduration=2.79019715 podStartE2EDuration="2.79019715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:03.711686082 +0000 UTC m=+1882.369782306" watchObservedRunningTime="2026-10-30 22:11:03.79019715 +0000 UTC m=+1882.448293377"
Oct 30 22:11:03.852141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:03.847665    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55" podStartSLOduration=2.847641382 podStartE2EDuration="2.847641382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:03.83291834 +0000 UTC m=+1882.491014564" watchObservedRunningTime="2026-10-30 22:11:03.847641382 +0000 UTC m=+1882.505737599"
Oct 30 22:11:04.535508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.535455    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"]
Oct 30 22:11:04.540203 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.540179    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.547870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.546926    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"payload-processing-external-model-e2e--a9381736-dockercfg-kqh8l\""
Oct 30 22:11:04.547870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.547020    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"agc-cred-3261044a2004\""
Oct 30 22:11:04.548061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.547922    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"payload-processing-external-model-plugins-e2e-extmod-b-26a128\""
Oct 30 22:11:04.548061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.548010    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"routing-overlay\""
Oct 30 22:11:04.548061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.548010    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"openshift-service-ca.crt\""
Oct 30 22:11:04.548061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.548032    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"kube-root-ca.crt\""
Oct 30 22:11:04.552841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.552818    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"]
Oct 30 22:11:04.694491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.678184    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2k4f\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-kube-api-access-j2k4f\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.694491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.678305    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.694491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.678330    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.694491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.678410    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.844548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.843762    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2k4f\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-kube-api-access-j2k4f\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.844548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.843907    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.844548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.843940    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.844548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.844028    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.851496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.845698    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.851496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.846641    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.946491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.943269    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.988138 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:04.978711    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2k4f\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-kube-api-access-j2k4f\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-59559n2br\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:04.988138 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:11:04.979738    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:11:05.162088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:05.162007    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:05.195318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:05.195284    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:11:05.920634 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:05.910891    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"]
Oct 30 22:11:06.036312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.026040    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:06.111840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.092907    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f0f5e2\""
Oct 30 22:11:06.116855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.116830    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"]
Oct 30 22:11:06.123981 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:11:06.123958    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fea9bb_c664_46d7_9d9f_224e228cf339.slice/crio-41046418ed29fcac2e997ce539dd02db6da3115644c080787d681513d39e71f7 WatchSource:0}: Error finding container 41046418ed29fcac2e997ce539dd02db6da3115644c080787d681513d39e71f7: Status 404 returned error can't find the container with id 41046418ed29fcac2e997ce539dd02db6da3115644c080787d681513d39e71f7
Oct 30 22:11:06.133855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.133568    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"]
Oct 30 22:11:06.236066 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.236037    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490fe5a2-3e4b-4867-b20f-124f062e0685-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs\" (UID: \"490fe5a2-3e4b-4867-b20f-124f062e0685\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:06.236677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.236649    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrc5\" (UniqueName: \"kubernetes.io/projected/490fe5a2-3e4b-4867-b20f-124f062e0685-kube-api-access-qdrc5\") pod \"payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs\" (UID: \"490fe5a2-3e4b-4867-b20f-124f062e0685\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:06.338188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.338139    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrc5\" (UniqueName: \"kubernetes.io/projected/490fe5a2-3e4b-4867-b20f-124f062e0685-kube-api-access-qdrc5\") pod \"payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs\" (UID: \"490fe5a2-3e4b-4867-b20f-124f062e0685\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:06.342490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.339095    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490fe5a2-3e4b-4867-b20f-124f062e0685-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs\" (UID: \"490fe5a2-3e4b-4867-b20f-124f062e0685\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:06.350518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.343494    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490fe5a2-3e4b-4867-b20f-124f062e0685-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs\" (UID: \"490fe5a2-3e4b-4867-b20f-124f062e0685\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:06.370198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.370159    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrc5\" (UniqueName: \"kubernetes.io/projected/490fe5a2-3e4b-4867-b20f-124f062e0685-kube-api-access-qdrc5\") pod \"payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs\" (UID: \"490fe5a2-3e4b-4867-b20f-124f062e0685\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:06.451955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:06.451860    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:07.041081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:07.041039    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"]
Oct 30 22:11:07.333490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:07.331973    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs" event={"ID":"490fe5a2-3e4b-4867-b20f-124f062e0685","Type":"ContainerStarted","Data":"73b105e0254394ed446e557f4a9009a93a13a30f5521a9cc4ebd477cb00666d7"}
Oct 30 22:11:07.342925 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:07.342876    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br" event={"ID":"97fea9bb-c664-46d7-9d9f-224e228cf339","Type":"ContainerStarted","Data":"0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd"}
Oct 30 22:11:07.343392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:07.343207    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br" event={"ID":"97fea9bb-c664-46d7-9d9f-224e228cf339","Type":"ContainerStarted","Data":"41046418ed29fcac2e997ce539dd02db6da3115644c080787d681513d39e71f7"}
Oct 30 22:11:07.352668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:07.352648    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:07.429179 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:07.429114    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br" podStartSLOduration=3.429092258 podStartE2EDuration="3.429092258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:07.38067628 +0000 UTC m=+1886.038772503" watchObservedRunningTime="2026-10-30 22:11:07.429092258 +0000 UTC m=+1886.087188482"
Oct 30 22:11:08.447154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:08.443816    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs" event={"ID":"490fe5a2-3e4b-4867-b20f-124f062e0685","Type":"ContainerStarted","Data":"301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7"}
Oct 30 22:11:08.447154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:08.443934    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:08.497493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:08.491972    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs" podStartSLOduration=3.491954377 podStartE2EDuration="3.491954377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:08.490269477 +0000 UTC m=+1887.148365702" watchObservedRunningTime="2026-10-30 22:11:08.491954377 +0000 UTC m=+1887.150050602"
Oct 30 22:11:11.335049 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:11.335007    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4" podUID="66562807-8fd2-448d-8187-95729caa75f7" containerName="main" containerID="cri-o://3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605" gracePeriod=45
Oct 30 22:11:12.683497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.658653    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:11:12.809997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.717235    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"]
Oct 30 22:11:12.809997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.718538    4282 cpu_manager.go:414] "containerMap: removing container" podUID="66562807-8fd2-448d-8187-95729caa75f7" containerName="main"
Oct 30 22:11:12.809997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.718561    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66562807-8fd2-448d-8187-95729caa75f7" containerName="main"
Oct 30 22:11:12.809997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.718864    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="66562807-8fd2-448d-8187-95729caa75f7" containerName="main"
Oct 30 22:11:12.809997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.759271    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-home\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-home\") pod \"66562807-8fd2-448d-8187-95729caa75f7\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") "
Oct 30 22:11:12.809997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.759324    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-tmp-dir\") pod \"66562807-8fd2-448d-8187-95729caa75f7\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") "
Oct 30 22:11:12.809997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.759357    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66562807-8fd2-448d-8187-95729caa75f7-kube-api-access-wdnnh\" (UniqueName: \"kubernetes.io/projected/66562807-8fd2-448d-8187-95729caa75f7-kube-api-access-wdnnh\") pod \"66562807-8fd2-448d-8187-95729caa75f7\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") "
Oct 30 22:11:12.814827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.813830    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-home" pod "66562807-8fd2-448d-8187-95729caa75f7" (UID: "66562807-8fd2-448d-8187-95729caa75f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:12.814827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.813900    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-tmp-dir" pod "66562807-8fd2-448d-8187-95729caa75f7" (UID: "66562807-8fd2-448d-8187-95729caa75f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:12.830432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.828788    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66562807-8fd2-448d-8187-95729caa75f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/66562807-8fd2-448d-8187-95729caa75f7-tls-certs\") pod \"66562807-8fd2-448d-8187-95729caa75f7\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") "
Oct 30 22:11:12.830432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.828888    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-model-cache\") pod \"66562807-8fd2-448d-8187-95729caa75f7\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") "
Oct 30 22:11:12.830432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.828958    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-dshm\") pod \"66562807-8fd2-448d-8187-95729caa75f7\" (UID: \"66562807-8fd2-448d-8187-95729caa75f7\") "
Oct 30 22:11:12.830432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.829978    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:12.830432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.830001    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:12.893904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.893866    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:12.939586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.939543    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"]
Oct 30 22:11:12.971511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.967563    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-model-cache" pod "66562807-8fd2-448d-8187-95729caa75f7" (UID: "66562807-8fd2-448d-8187-95729caa75f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:12.972002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:12.971980    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a5f192-openshift-default-dockercfg-829z5\""
Oct 30 22:11:13.218344 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.218303    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:13.218734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.218706    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66562807-8fd2-448d-8187-95729caa75f7-tls-certs" pod "66562807-8fd2-448d-8187-95729caa75f7" (UID: "66562807-8fd2-448d-8187-95729caa75f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:13.271291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.270735    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4" podUID="66562807-8fd2-448d-8187-95729caa75f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.174:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:13.348084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.347994    4282 generic.go:356] "Generic (PLEG): container finished" podID="66562807-8fd2-448d-8187-95729caa75f7" containerID="3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605" exitCode=0
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.350982    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqkg\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-kube-api-access-ckqkg\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.353931    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-data\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.354058    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-envoy\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.354154    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774525b8-c126-4535-a248-0b63d68e599a-istio-podinfo\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.354204    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774525b8-c126-4535-a248-0b63d68e599a-istiod-ca-cert\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.354239    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.354309    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-istio-token\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.354414    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-certs\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.354767    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-credential-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.355490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.355278    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66562807-8fd2-448d-8187-95729caa75f7-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:13.375493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.365329    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"
Oct 30 22:11:13.375739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.375704    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4" event={"ID":"66562807-8fd2-448d-8187-95729caa75f7","Type":"ContainerDied","Data":"3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605"}
Oct 30 22:11:13.379491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.375882    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4" event={"ID":"66562807-8fd2-448d-8187-95729caa75f7","Type":"ContainerDied","Data":"4babbb721a68a705050e2922028b0a1708f3f326287046baa65d53ad99e66aa1"}
Oct 30 22:11:13.379713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.379680    4282 scope.go:122] "RemoveContainer" containerID="3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605"
Oct 30 22:11:13.414129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.412959    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66562807-8fd2-448d-8187-95729caa75f7-kube-api-access-wdnnh" pod "66562807-8fd2-448d-8187-95729caa75f7" (UID: "66562807-8fd2-448d-8187-95729caa75f7"). InnerVolumeSpecName "kube-api-access-wdnnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:13.414129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.413790    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-dshm" pod "66562807-8fd2-448d-8187-95729caa75f7" (UID: "66562807-8fd2-448d-8187-95729caa75f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:13.483806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.483761    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774525b8-c126-4535-a248-0b63d68e599a-istio-podinfo\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.484746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.484719    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774525b8-c126-4535-a248-0b63d68e599a-istiod-ca-cert\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.484947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.484927    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.485112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.485096    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-istio-token\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.485299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.485283    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-certs\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.485539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.485518    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-credential-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.485812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.485789    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqkg\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-kube-api-access-ckqkg\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.486010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.485992    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-data\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.486220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.486203    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-envoy\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.486569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.486550    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66562807-8fd2-448d-8187-95729caa75f7-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:13.486706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.486692    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdnnh\" (UniqueName: \"kubernetes.io/projected/66562807-8fd2-448d-8187-95729caa75f7-kube-api-access-wdnnh\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:13.487579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.487557    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-certs\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.488397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.488375    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774525b8-c126-4535-a248-0b63d68e599a-istiod-ca-cert\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.490435 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.489004    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774525b8-c126-4535-a248-0b63d68e599a-istio-podinfo\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.490624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.489631    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-credential-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.490811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.489876    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-socket\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.490935 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.490120    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-data\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.504283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.504254    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-envoy\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.546425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.546340    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-istio-token\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.546661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.546512    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqkg\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-kube-api-access-ckqkg\") pod \"e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") " pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.549324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.549049    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:13.674493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.670361    4282 scope.go:122] "RemoveContainer" containerID="3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605"
Oct 30 22:11:13.737283 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:11:13.713798    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605\": container with ID starting with 3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605 not found: ID does not exist" containerID="3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605"
Oct 30 22:11:13.737283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.713856    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605"} err="failed to get container status \"3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605\": rpc error: code = NotFound desc = could not find container \"3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605\": container with ID starting with 3e288a7759d0ec51c0656a7f8a455a4870fca7d2b2a33dd0fbcdb21a79f18605 not found: ID does not exist"
Oct 30 22:11:13.913725 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.913634    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:13.985575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:13.985539    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"]
Oct 30 22:11:14.041933 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:14.041890    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5313f0/auth-test-model-5313f0-kserve-54c5885fcc-hnkj4"]
Oct 30 22:11:14.782990 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:11:14.782958    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774525b8_c126_4535_a248_0b63d68e599a.slice/crio-3b4c4f1d0ca1d4d5645cb92c1639c69a7cd5e644baaa5a5b541eff835b88e7f9 WatchSource:0}: Error finding container 3b4c4f1d0ca1d4d5645cb92c1639c69a7cd5e644baaa5a5b541eff835b88e7f9: Status 404 returned error can't find the container with id 3b4c4f1d0ca1d4d5645cb92c1639c69a7cd5e644baaa5a5b541eff835b88e7f9
Oct 30 22:11:14.825080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:14.825029    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:14.825374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:14.825347    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:14.825578 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:14.825556    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:14.893794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:14.893764    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:11:14.908495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:14.895685    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"]
Oct 30 22:11:15.469875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:15.469843    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66562807-8fd2-448d-8187-95729caa75f7" path="/var/lib/kubelet/pods/66562807-8fd2-448d-8187-95729caa75f7/volumes"
Oct 30 22:11:15.470330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:15.470304    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" event={"ID":"774525b8-c126-4535-a248-0b63d68e599a","Type":"ContainerStarted","Data":"9d893cc82712c7ab4628cbda09cdc6d50e0e1db806d28d87dd88823edab40961"}
Oct 30 22:11:15.470412 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:15.470361    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" event={"ID":"774525b8-c126-4535-a248-0b63d68e599a","Type":"ContainerStarted","Data":"3b4c4f1d0ca1d4d5645cb92c1639c69a7cd5e644baaa5a5b541eff835b88e7f9"}
Oct 30 22:11:15.523075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:15.522961    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" podStartSLOduration=3.522939068 podStartE2EDuration="3.522939068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:15.498882717 +0000 UTC m=+1894.156978941" watchObservedRunningTime="2026-10-30 22:11:15.522939068 +0000 UTC m=+1894.181035295"
Oct 30 22:11:15.555094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:15.550233    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:16.157299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:16.157268    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:11:16.557364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:16.557333    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:17.482814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:17.482712    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:17.484801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:17.484779    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:11:18.844491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:18.836360    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:18.844491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:18.836420    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:19.450492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:19.450425    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:11:19.452183 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:19.452156    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:11:20.158142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.158108    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"]
Oct 30 22:11:20.216964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.216932    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"]
Oct 30 22:11:20.221292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.221268    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:20.282737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.282328    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:20.282737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.282411    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:20.368491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.336280    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae82a63f-9f4c-4a2a-b713-127e52277966-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp\" (UID: \"ae82a63f-9f4c-4a2a-b713-127e52277966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:20.368491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.336600    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqqlj\" (UniqueName: \"kubernetes.io/projected/ae82a63f-9f4c-4a2a-b713-127e52277966-kube-api-access-rqqlj\") pod \"payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp\" (UID: \"ae82a63f-9f4c-4a2a-b713-127e52277966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:20.403508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.385813    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a5f192-dockercfg-db87c\""
Oct 30 22:11:20.403508 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.385921    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a5f192\""
Oct 30 22:11:20.420073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.411533    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"]
Oct 30 22:11:20.420073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.411663    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:20.461606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.457584    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"]
Oct 30 22:11:20.491570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.489854    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqqlj\" (UniqueName: \"kubernetes.io/projected/ae82a63f-9f4c-4a2a-b713-127e52277966-kube-api-access-rqqlj\") pod \"payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp\" (UID: \"ae82a63f-9f4c-4a2a-b713-127e52277966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:20.491570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.489909    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx2n2\" (UniqueName: \"kubernetes.io/projected/0688b402-ba16-4049-a38e-ece12d389c7d-kube-api-access-qx2n2\") pod \"payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs\" (UID: \"0688b402-ba16-4049-a38e-ece12d389c7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:20.491570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.490136    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae82a63f-9f4c-4a2a-b713-127e52277966-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp\" (UID: \"ae82a63f-9f4c-4a2a-b713-127e52277966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:20.491570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.490199    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0688b402-ba16-4049-a38e-ece12d389c7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs\" (UID: \"0688b402-ba16-4049-a38e-ece12d389c7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:20.491570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.491517    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae82a63f-9f4c-4a2a-b713-127e52277966-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp\" (UID: \"ae82a63f-9f4c-4a2a-b713-127e52277966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:20.598370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.598324    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx2n2\" (UniqueName: \"kubernetes.io/projected/0688b402-ba16-4049-a38e-ece12d389c7d-kube-api-access-qx2n2\") pod \"payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs\" (UID: \"0688b402-ba16-4049-a38e-ece12d389c7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:20.598755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.598732    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0688b402-ba16-4049-a38e-ece12d389c7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs\" (UID: \"0688b402-ba16-4049-a38e-ece12d389c7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:20.633493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.618092    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0688b402-ba16-4049-a38e-ece12d389c7d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs\" (UID: \"0688b402-ba16-4049-a38e-ece12d389c7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:20.652097 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.646138    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqqlj\" (UniqueName: \"kubernetes.io/projected/ae82a63f-9f4c-4a2a-b713-127e52277966-kube-api-access-rqqlj\") pod \"payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp\" (UID: \"ae82a63f-9f4c-4a2a-b713-127e52277966\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:20.655055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.654969    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx2n2\" (UniqueName: \"kubernetes.io/projected/0688b402-ba16-4049-a38e-ece12d389c7d-kube-api-access-qx2n2\") pod \"payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs\" (UID: \"0688b402-ba16-4049-a38e-ece12d389c7d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:20.739889 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.739801    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:20.779797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:20.779767    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:21.117397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.117349    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"]
Oct 30 22:11:21.117956 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:11:21.117903    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae82a63f_9f4c_4a2a_b713_127e52277966.slice/crio-0994c659ca1245f2a79e847bc35dd865c28f47b56cd68f6edced6350447040dc WatchSource:0}: Error finding container 0994c659ca1245f2a79e847bc35dd865c28f47b56cd68f6edced6350447040dc: Status 404 returned error can't find the container with id 0994c659ca1245f2a79e847bc35dd865c28f47b56cd68f6edced6350447040dc
Oct 30 22:11:21.121991 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:11:21.121958    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0688b402_ba16_4049_a38e_ece12d389c7d.slice/crio-24f118d0b62df7631104226a993b230342eca235f5f4363f5c64aaeb6f92c5fa WatchSource:0}: Error finding container 24f118d0b62df7631104226a993b230342eca235f5f4363f5c64aaeb6f92c5fa: Status 404 returned error can't find the container with id 24f118d0b62df7631104226a993b230342eca235f5f4363f5c64aaeb6f92c5fa
Oct 30 22:11:21.125244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.125212    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"]
Oct 30 22:11:22.419842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.694778    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp" event={"ID":"ae82a63f-9f4c-4a2a-b713-127e52277966","Type":"ContainerStarted","Data":"6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4"}
Oct 30 22:11:22.419842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.694840    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp" event={"ID":"ae82a63f-9f4c-4a2a-b713-127e52277966","Type":"ContainerStarted","Data":"0994c659ca1245f2a79e847bc35dd865c28f47b56cd68f6edced6350447040dc"}
Oct 30 22:11:22.419842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.694868    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:22.419842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.696951    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs" event={"ID":"0688b402-ba16-4049-a38e-ece12d389c7d","Type":"ContainerStarted","Data":"86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03"}
Oct 30 22:11:22.419842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.696990    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs" event={"ID":"0688b402-ba16-4049-a38e-ece12d389c7d","Type":"ContainerStarted","Data":"24f118d0b62df7631104226a993b230342eca235f5f4363f5c64aaeb6f92c5fa"}
Oct 30 22:11:22.419842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.697047    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:22.419842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.713061    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp" podStartSLOduration=1.713044491 podStartE2EDuration="1.713044491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:21.711087345 +0000 UTC m=+1900.369183568" watchObservedRunningTime="2026-10-30 22:11:21.713044491 +0000 UTC m=+1900.371140714"
Oct 30 22:11:22.419842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:21.732018    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs" podStartSLOduration=1.732005772 podStartE2EDuration="1.732005772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:21.727335687 +0000 UTC m=+1900.385431910" watchObservedRunningTime="2026-10-30 22:11:21.732005772 +0000 UTC m=+1900.390101996"
Oct 30 22:11:22.707146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.707057    4282 generic.go:356] "Generic (PLEG): container finished" podID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerID="df7e3128ba0207a81ce8384143609033ee1230c7855c8d0b1a0cab08bd18d311" exitCode=0
Oct 30 22:11:22.707146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.707125    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" event={"ID":"3e713fa3-0235-4cb6-aea0-65fa0591293d","Type":"ContainerDied","Data":"df7e3128ba0207a81ce8384143609033ee1230c7855c8d0b1a0cab08bd18d311"}
Oct 30 22:11:22.707611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.707596    4282 scope.go:122] "RemoveContainer" containerID="df7e3128ba0207a81ce8384143609033ee1230c7855c8d0b1a0cab08bd18d311"
Oct 30 22:11:22.708910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.708891    4282 generic.go:356] "Generic (PLEG): container finished" podID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerID="4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d" exitCode=0
Oct 30 22:11:22.708997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.708965    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" event={"ID":"bd34ab38-6807-4efc-961f-6b10adb0f268","Type":"ContainerDied","Data":"4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d"}
Oct 30 22:11:22.709436 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.709421    4282 scope.go:122] "RemoveContainer" containerID="4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d"
Oct 30 22:11:22.709694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.709650    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:22.709804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.709749    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:22.709873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.709798    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:22.711111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.711091    4282 generic.go:356] "Generic (PLEG): container finished" podID="71d838db-a10d-4705-8c50-09e5ce26689a" containerID="92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f" exitCode=0
Oct 30 22:11:22.711224 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.711152    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" event={"ID":"71d838db-a10d-4705-8c50-09e5ce26689a","Type":"ContainerDied","Data":"92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f"}
Oct 30 22:11:22.711545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.711528    4282 scope.go:122] "RemoveContainer" containerID="92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f"
Oct 30 22:11:22.711853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.711829    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:22.711907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.711887    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:22.711955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.711919    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:22.713996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.713970    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:22.714079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.714051    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:22.714142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:22.714096    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:23.724041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:23.723988    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" event={"ID":"71d838db-a10d-4705-8c50-09e5ce26689a","Type":"ContainerStarted","Data":"d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b"}
Oct 30 22:11:23.727167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:23.727133    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" event={"ID":"3e713fa3-0235-4cb6-aea0-65fa0591293d","Type":"ContainerStarted","Data":"e4f1bec4abda328466055660d7a30a9e54a6efe88ef09edc271a037ae923a851"}
Oct 30 22:11:23.730292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:23.730262    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" event={"ID":"bd34ab38-6807-4efc-961f-6b10adb0f268","Type":"ContainerStarted","Data":"90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d"}
Oct 30 22:11:23.934492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:23.933910    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:11:24.012547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.008751    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:11:24.012547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.008880    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:11:24.012547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.008925    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:11:24.563185 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.557891    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:11:24.577491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.567627    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:11:24.577491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.567690    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 22:11:24.731382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.730945    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:11:24.731382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.731080    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:11:24.731382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:24.731125    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 22:11:25.011854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:25.011815    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:11:25.012143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:25.012115    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:11:25.572494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:25.571593    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:11:25.572494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:25.571655    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 22:11:25.716503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:25.716442    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:11:25.716795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:25.716747    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 22:11:26.013495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:26.009907    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:11:26.013495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:26.009969    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:11:26.555495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:26.551612    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:11:26.555495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:26.551676    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 22:11:26.714072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:26.714033    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:11:26.714422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:26.714394    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 22:11:27.024494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:27.022893    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:11:27.024494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:27.022954    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:11:27.569854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:27.552861    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:11:27.600648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:27.600605    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 22:11:27.735785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:27.735736    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:11:27.736086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:27.736051    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 22:11:28.023622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:28.023587    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:11:28.023868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:28.023845    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:11:28.574495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:28.573593    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:11:28.574495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:28.573654    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 22:11:28.719506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:28.716301    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:11:28.719506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:28.716358    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 22:11:29.023493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:29.022879    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:11:29.023493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:29.022937    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:11:29.564666 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:29.564630    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:11:29.565428 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:29.565394    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Oct 30 22:11:29.734497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:29.730583    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body=
Oct 30 22:11:29.734497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:29.730655    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused"
Oct 30 22:11:30.007971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:30.007936    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 22:11:30.008374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:30.008257    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 22:11:30.555727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:30.555691    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:11:30.556166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:30.556146    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:11:30.556845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:30.556823    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:11:30.775535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:30.770316    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:11:30.782489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:30.781963    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:11:30.797481 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:30.791970    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:11:31.013619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:31.013590    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:11:31.014804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:31.014779    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:11:32.714648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:32.714623    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:11:32.715055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:32.714695    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:11:34.758378 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:11:34.758341    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:11:36.814931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:36.814896    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:11:45.844147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.844107    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"]
Oct 30 22:11:45.855728 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.855700    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.862170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.862142    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"]
Oct 30 22:11:45.862374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.862260    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f0f5e2\"/\"autores-model-f0f5e2-kserve-self-signed-certs\""
Oct 30 22:11:45.864926 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.864907    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f0f5e2\"/\"default-dockercfg-t4sgl\""
Oct 30 22:11:45.865937 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.865919    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f0f5e2\"/\"openshift-service-ca.crt\""
Oct 30 22:11:45.866227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.866183    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f0f5e2\"/\"kube-root-ca.crt\""
Oct 30 22:11:45.870894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.870286    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hxz\" (UniqueName: \"kubernetes.io/projected/ff7d5fbe-ef5a-4a89-81a2-6205da421739-kube-api-access-l5hxz\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.870894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.870407    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tls-certs\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.870894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.870446    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-dshm\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.870894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.870537    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-model-cache\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.870894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.870593    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-home\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.870894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.870716    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tmp-dir\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.972755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.972713    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hxz\" (UniqueName: \"kubernetes.io/projected/ff7d5fbe-ef5a-4a89-81a2-6205da421739-kube-api-access-l5hxz\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.973132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.973099    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tls-certs\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.973276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.973156    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-dshm\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.973276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.973228    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-model-cache\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.973429 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.973300    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-home\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.973505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.973432    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tmp-dir\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.973942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.973919    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tmp-dir\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.974115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.974051    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-home\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.974218 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.974101    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-model-cache\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.976121 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.976100    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tls-certs\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.976541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.976519    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-dshm\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:45.987033 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:45.987009    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hxz\" (UniqueName: \"kubernetes.io/projected/ff7d5fbe-ef5a-4a89-81a2-6205da421739-kube-api-access-l5hxz\") pod \"autores-model-f0f5e2-kserve-75fd568d45-vpprm\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") " pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:46.177275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:46.177183    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:46.534669 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:11:46.534638    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff7d5fbe_ef5a_4a89_81a2_6205da421739.slice/crio-354318c272440af89ddbf08717c49407db0e6feaec023d0dc723143591d87da4 WatchSource:0}: Error finding container 354318c272440af89ddbf08717c49407db0e6feaec023d0dc723143591d87da4: Status 404 returned error can't find the container with id 354318c272440af89ddbf08717c49407db0e6feaec023d0dc723143591d87da4
Oct 30 22:11:46.534911 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:46.534888    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"]
Oct 30 22:11:47.328730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:47.326548    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm" event={"ID":"ff7d5fbe-ef5a-4a89-81a2-6205da421739","Type":"ContainerStarted","Data":"4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de"}
Oct 30 22:11:47.328730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:47.326609    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm" event={"ID":"ff7d5fbe-ef5a-4a89-81a2-6205da421739","Type":"ContainerStarted","Data":"354318c272440af89ddbf08717c49407db0e6feaec023d0dc723143591d87da4"}
Oct 30 22:11:47.347108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:47.347049    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm" podStartSLOduration=2.3470344389999998 podStartE2EDuration="2.347034439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:47.344702603 +0000 UTC m=+1926.002798827" watchObservedRunningTime="2026-10-30 22:11:47.347034439 +0000 UTC m=+1926.005130696"
Oct 30 22:11:48.177567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:48.177534    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:48.608661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:48.608581    4282 patch_prober.go:29] interesting pod/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:48.608661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:48.608626    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:50.354616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:50.354578    4282 generic.go:356] "Generic (PLEG): container finished" podID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerID="24cf27d0c576a34112da182a097e34d04e21d6885d1492457d223854b848b968" exitCode=0
Oct 30 22:11:50.355129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:50.354645    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" event={"ID":"f021a87d-4a02-4e67-b8c5-1c23d125d78a","Type":"ContainerDied","Data":"24cf27d0c576a34112da182a097e34d04e21d6885d1492457d223854b848b968"}
Oct 30 22:11:50.355257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:50.355243    4282 scope.go:122] "RemoveContainer" containerID="24cf27d0c576a34112da182a097e34d04e21d6885d1492457d223854b848b968"
Oct 30 22:11:50.357691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:50.357659    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:50.357827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:50.357749    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:50.357827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:50.357800    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:11:51.383212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:51.383176    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" event={"ID":"f021a87d-4a02-4e67-b8c5-1c23d125d78a","Type":"ContainerStarted","Data":"d11a2799578fbb09f697f6cb6fabafc194a5bfb4d1b69b568ca3bb9288751122"}
Oct 30 22:11:52.376497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:52.375649    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:52.376497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:52.376041    4282 patch_prober.go:29] interesting pod/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:11:52.376497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:52.376093    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:11:53.360900 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:53.360813    4282 patch_prober.go:29] interesting pod/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:11:53.361596 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:53.361561    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:11:54.368493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:54.362112    4282 patch_prober.go:29] interesting pod/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:11:54.368493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:54.362185    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:11:55.358833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:55.358799    4282 patch_prober.go:29] interesting pod/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:11:55.359143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:55.359110    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:11:56.179501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:56.178498    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:56.246416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:56.246384    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:56.356258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:56.355820    4282 patch_prober.go:29] interesting pod/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 22:11:56.356258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:56.355889    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 22:11:56.578823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:56.578720    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:11:57.357075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:57.357045    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:57.357620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:57.357448    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:11:57.358071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:11:57.358051    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:12:01.804183 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:01.804145    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"]
Oct 30 22:12:04.650723 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:12:04.650689    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:12:12.152537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:12.152496    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"]
Oct 30 22:12:16.497043 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:16.497009    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:12:16.841168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:16.841069    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm" podUID="ff7d5fbe-ef5a-4a89-81a2-6205da421739" containerName="main" containerID="cri-o://4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de" gracePeriod=45
Oct 30 22:12:16.949815 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:16.949782    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm" podUID="ff7d5fbe-ef5a-4a89-81a2-6205da421739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.214:8000/ready\": EOF"
Oct 30 22:12:17.143928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.143904    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:12:17.152992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.152963    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs" podUID="490fe5a2-3e4b-4867-b20f-124f062e0685" containerName="payload-pre-processing" containerID="cri-o://301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7" gracePeriod=25
Oct 30 22:12:17.257928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.257897    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-home\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-home\") pod \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") "
Oct 30 22:12:17.258110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.257953    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-dshm\") pod \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") "
Oct 30 22:12:17.258110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.258059    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-model-cache\") pod \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") "
Oct 30 22:12:17.258246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.258109    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tls-certs\") pod \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") "
Oct 30 22:12:17.258246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.258121    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-home" pod "ff7d5fbe-ef5a-4a89-81a2-6205da421739" (UID: "ff7d5fbe-ef5a-4a89-81a2-6205da421739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:17.258358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.258245    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff7d5fbe-ef5a-4a89-81a2-6205da421739-kube-api-access-l5hxz\" (UniqueName: \"kubernetes.io/projected/ff7d5fbe-ef5a-4a89-81a2-6205da421739-kube-api-access-l5hxz\") pod \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") "
Oct 30 22:12:17.258358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.258341    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-model-cache" pod "ff7d5fbe-ef5a-4a89-81a2-6205da421739" (UID: "ff7d5fbe-ef5a-4a89-81a2-6205da421739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:17.258530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.258510    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tmp-dir\") pod \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\" (UID: \"ff7d5fbe-ef5a-4a89-81a2-6205da421739\") "
Oct 30 22:12:17.259139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.258950    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tmp-dir" pod "ff7d5fbe-ef5a-4a89-81a2-6205da421739" (UID: "ff7d5fbe-ef5a-4a89-81a2-6205da421739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:17.259550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.259530    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.259647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.259555    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.259647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.259570    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.260664 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.260639    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-dshm" pod "ff7d5fbe-ef5a-4a89-81a2-6205da421739" (UID: "ff7d5fbe-ef5a-4a89-81a2-6205da421739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:17.260941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.260897    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tls-certs" pod "ff7d5fbe-ef5a-4a89-81a2-6205da421739" (UID: "ff7d5fbe-ef5a-4a89-81a2-6205da421739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:17.261110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.261089    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff7d5fbe-ef5a-4a89-81a2-6205da421739-kube-api-access-l5hxz" pod "ff7d5fbe-ef5a-4a89-81a2-6205da421739" (UID: "ff7d5fbe-ef5a-4a89-81a2-6205da421739"). InnerVolumeSpecName "kube-api-access-l5hxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:17.361095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.361020    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff7d5fbe-ef5a-4a89-81a2-6205da421739-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.361095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.361049    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff7d5fbe-ef5a-4a89-81a2-6205da421739-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.361095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.361065    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5hxz\" (UniqueName: \"kubernetes.io/projected/ff7d5fbe-ef5a-4a89-81a2-6205da421739-kube-api-access-l5hxz\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.416455 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.416420    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:12:17.564126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.564087    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490fe5a2-3e4b-4867-b20f-124f062e0685-kube-api-access-qdrc5\" (UniqueName: \"kubernetes.io/projected/490fe5a2-3e4b-4867-b20f-124f062e0685-kube-api-access-qdrc5\") pod \"490fe5a2-3e4b-4867-b20f-124f062e0685\" (UID: \"490fe5a2-3e4b-4867-b20f-124f062e0685\") "
Oct 30 22:12:17.564535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.564291    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490fe5a2-3e4b-4867-b20f-124f062e0685-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490fe5a2-3e4b-4867-b20f-124f062e0685-plugins-config-volume\") pod \"490fe5a2-3e4b-4867-b20f-124f062e0685\" (UID: \"490fe5a2-3e4b-4867-b20f-124f062e0685\") "
Oct 30 22:12:17.564759 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.564734    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490fe5a2-3e4b-4867-b20f-124f062e0685-plugins-config-volume" pod "490fe5a2-3e4b-4867-b20f-124f062e0685" (UID: "490fe5a2-3e4b-4867-b20f-124f062e0685"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:17.565092 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.565073    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/490fe5a2-3e4b-4867-b20f-124f062e0685-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.566456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.566431    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490fe5a2-3e4b-4867-b20f-124f062e0685-kube-api-access-qdrc5" pod "490fe5a2-3e4b-4867-b20f-124f062e0685" (UID: "490fe5a2-3e4b-4867-b20f-124f062e0685"). InnerVolumeSpecName "kube-api-access-qdrc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:17.672286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.672241    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdrc5\" (UniqueName: \"kubernetes.io/projected/490fe5a2-3e4b-4867-b20f-124f062e0685-kube-api-access-qdrc5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.788139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.788101    4282 generic.go:356] "Generic (PLEG): container finished" podID="490fe5a2-3e4b-4867-b20f-124f062e0685" containerID="301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7" exitCode=0
Oct 30 22:12:17.788314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.788154    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"
Oct 30 22:12:17.788314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.788184    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs" event={"ID":"490fe5a2-3e4b-4867-b20f-124f062e0685","Type":"ContainerDied","Data":"301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7"}
Oct 30 22:12:17.788314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.788246    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs" event={"ID":"490fe5a2-3e4b-4867-b20f-124f062e0685","Type":"ContainerDied","Data":"73b105e0254394ed446e557f4a9009a93a13a30f5521a9cc4ebd477cb00666d7"}
Oct 30 22:12:17.788314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.788269    4282 scope.go:122] "RemoveContainer" containerID="301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7"
Oct 30 22:12:17.789956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.789934    4282 generic.go:356] "Generic (PLEG): container finished" podID="ff7d5fbe-ef5a-4a89-81a2-6205da421739" containerID="4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de" exitCode=0
Oct 30 22:12:17.790077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.789987    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"
Oct 30 22:12:17.790077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.790009    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm" event={"ID":"ff7d5fbe-ef5a-4a89-81a2-6205da421739","Type":"ContainerDied","Data":"4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de"}
Oct 30 22:12:17.790077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.790034    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm" event={"ID":"ff7d5fbe-ef5a-4a89-81a2-6205da421739","Type":"ContainerDied","Data":"354318c272440af89ddbf08717c49407db0e6feaec023d0dc723143591d87da4"}
Oct 30 22:12:17.800431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.800412    4282 scope.go:122] "RemoveContainer" containerID="301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7"
Oct 30 22:12:17.800748 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:12:17.800729    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7\": container with ID starting with 301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7 not found: ID does not exist" containerID="301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7"
Oct 30 22:12:17.800807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.800758    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7"} err="failed to get container status \"301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7\": rpc error: code = NotFound desc = could not find container \"301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7\": container with ID starting with 301cc0ba5aa889fe5af20666e805cdb0947b605c0ef45553a6dc8c985d6e7bc7 not found: ID does not exist"
Oct 30 22:12:17.800807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.800774    4282 scope.go:122] "RemoveContainer" containerID="4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de"
Oct 30 22:12:17.810776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.810752    4282 scope.go:122] "RemoveContainer" containerID="4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de"
Oct 30 22:12:17.811045 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:12:17.811027    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de\": container with ID starting with 4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de not found: ID does not exist" containerID="4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de"
Oct 30 22:12:17.811106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.811052    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de"} err="failed to get container status \"4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de\": rpc error: code = NotFound desc = could not find container \"4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de\": container with ID starting with 4be9aa54131ce28f4385f834bd83766732fba2ffd373254a5e5438863ad661de not found: ID does not exist"
Oct 30 22:12:17.817656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.817633    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"]
Oct 30 22:12:17.821792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.821767    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f0f5e2-7c64946795-2sxhs"]
Oct 30 22:12:17.830789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.830770    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"]
Oct 30 22:12:17.836260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:17.836238    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f0f5e2/autores-model-f0f5e2-kserve-75fd568d45-vpprm"]
Oct 30 22:12:19.484654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:19.484620    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490fe5a2-3e4b-4867-b20f-124f062e0685" path="/var/lib/kubelet/pods/490fe5a2-3e4b-4867-b20f-124f062e0685/volumes"
Oct 30 22:12:19.485250 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:19.485233    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff7d5fbe-ef5a-4a89-81a2-6205da421739" path="/var/lib/kubelet/pods/ff7d5fbe-ef5a-4a89-81a2-6205da421739/volumes"
Oct 30 22:12:22.895501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:22.895445    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"]
Oct 30 22:12:22.896285 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:22.896248    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy" containerID="cri-o://d11a2799578fbb09f697f6cb6fabafc194a5bfb4d1b69b568ca3bb9288751122" gracePeriod=30
Oct 30 22:12:27.360168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:27.360128    4282 patch_prober.go:29] interesting pod/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:12:27.360637 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:27.360190    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:12:27.958796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:27.958763    4282 generic.go:356] "Generic (PLEG): container finished" podID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerID="d11a2799578fbb09f697f6cb6fabafc194a5bfb4d1b69b568ca3bb9288751122" exitCode=0
Oct 30 22:12:27.958926 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:27.958827    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" event={"ID":"f021a87d-4a02-4e67-b8c5-1c23d125d78a","Type":"ContainerDied","Data":"d11a2799578fbb09f697f6cb6fabafc194a5bfb4d1b69b568ca3bb9288751122"}
Oct 30 22:12:27.958926 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:27.958893    4282 scope.go:122] "RemoveContainer" containerID="24cf27d0c576a34112da182a097e34d04e21d6885d1492457d223854b848b968"
Oct 30 22:12:28.692635 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.692554    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:12:28.838367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838326    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istiod-ca-cert\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.838669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838459    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-certs\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.838669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838533    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-podinfo\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.838669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838577    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-data\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.838669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838623    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-token\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-token\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.840896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838690    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-envoy\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.840896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838740    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-kube-api-access-6gcpp\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.840896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838806    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-socket\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.840896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.838939    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-credential-socket\") pod \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\" (UID: \"f021a87d-4a02-4e67-b8c5-1c23d125d78a\") "
Oct 30 22:12:28.844901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.842913    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-data" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:28.844901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.843274    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istiod-ca-cert" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:28.844901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.843515    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-certs" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:28.844901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.844543    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-socket" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:28.844901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.844731    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-credential-socket" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:28.848655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.845432    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.848655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.845457    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.848655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.845488    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.848655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.845503    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.848655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.845516    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.856807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.856754    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-envoy" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:12:28.857162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.857130    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-kube-api-access-6gcpp" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "kube-api-access-6gcpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:28.857305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.857177    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-token" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:28.858934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.858908    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-podinfo" pod "f021a87d-4a02-4e67-b8c5-1c23d125d78a" (UID: "f021a87d-4a02-4e67-b8c5-1c23d125d78a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:12:28.946786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.946722    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.946786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.946752    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.946786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.946765    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f021a87d-4a02-4e67-b8c5-1c23d125d78a-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.946786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.946779    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcpp\" (UniqueName: \"kubernetes.io/projected/f021a87d-4a02-4e67-b8c5-1c23d125d78a-kube-api-access-6gcpp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:28.970008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.969978    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"
Oct 30 22:12:28.970154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.969997    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9" event={"ID":"f021a87d-4a02-4e67-b8c5-1c23d125d78a","Type":"ContainerDied","Data":"a589b0d9e88c575029e885c7481975afbe3863a4a174ecab4c3a79c0aec51c25"}
Oct 30 22:12:28.970154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:28.970095    4282 scope.go:122] "RemoveContainer" containerID="d11a2799578fbb09f697f6cb6fabafc194a5bfb4d1b69b568ca3bb9288751122"
Oct 30 22:12:29.002080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.002054    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"]
Oct 30 22:12:29.014014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.013987    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-f0f5e2-openshift-default-8dd967bcf-blrs9"]
Oct 30 22:12:29.467200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.467170    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" path="/var/lib/kubelet/pods/f021a87d-4a02-4e67-b8c5-1c23d125d78a/volumes"
Oct 30 22:12:29.980990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.980959    4282 generic.go:356] "Generic (PLEG): container finished" podID="774525b8-c126-4535-a248-0b63d68e599a" containerID="9d893cc82712c7ab4628cbda09cdc6d50e0e1db806d28d87dd88823edab40961" exitCode=0
Oct 30 22:12:29.980990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.980992    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" event={"ID":"774525b8-c126-4535-a248-0b63d68e599a","Type":"ContainerDied","Data":"9d893cc82712c7ab4628cbda09cdc6d50e0e1db806d28d87dd88823edab40961"}
Oct 30 22:12:29.981582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.981384    4282 scope.go:122] "RemoveContainer" containerID="9d893cc82712c7ab4628cbda09cdc6d50e0e1db806d28d87dd88823edab40961"
Oct 30 22:12:29.983689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.983651    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:12:29.983782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.983760    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:12:29.983867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:29.983839    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:12:30.295824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.295741    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"]
Oct 30 22:12:30.297023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.296994    4282 cpu_manager.go:414] "containerMap: removing container" podUID="490fe5a2-3e4b-4867-b20f-124f062e0685" containerName="payload-pre-processing"
Oct 30 22:12:30.297023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297022    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490fe5a2-3e4b-4867-b20f-124f062e0685" containerName="payload-pre-processing"
Oct 30 22:12:30.297231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297037    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy"
Oct 30 22:12:30.297231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297046    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy"
Oct 30 22:12:30.297231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297062    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ff7d5fbe-ef5a-4a89-81a2-6205da421739" containerName="main"
Oct 30 22:12:30.297231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297070    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff7d5fbe-ef5a-4a89-81a2-6205da421739" containerName="main"
Oct 30 22:12:30.297231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297081    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy"
Oct 30 22:12:30.297231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297088    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy"
Oct 30 22:12:30.297595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297309    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="490fe5a2-3e4b-4867-b20f-124f062e0685" containerName="payload-pre-processing"
Oct 30 22:12:30.297595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297323    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy"
Oct 30 22:12:30.297595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297331    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff7d5fbe-ef5a-4a89-81a2-6205da421739" containerName="main"
Oct 30 22:12:30.297595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.297338    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f021a87d-4a02-4e67-b8c5-1c23d125d78a" containerName="istio-proxy"
Oct 30 22:12:30.302167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.302145    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"]
Oct 30 22:12:30.302310 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.302254    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:30.309621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.309597    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:30.311574 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.311550    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"]
Oct 30 22:12:30.318423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.318398    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6f8222\""
Oct 30 22:12:30.325810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.325782    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"]
Oct 30 22:12:30.387281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.387075    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxt7c\" (UniqueName: \"kubernetes.io/projected/ab57aefb-f4f7-4061-94f7-8effed875817-kube-api-access-qxt7c\") pod \"payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f\" (UID: \"ab57aefb-f4f7-4061-94f7-8effed875817\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:30.387281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.387273    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1913325b-12d7-4350-bb11-3207d9f870a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b\" (UID: \"1913325b-12d7-4350-bb11-3207d9f870a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:30.387587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.387372    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjdws\" (UniqueName: \"kubernetes.io/projected/1913325b-12d7-4350-bb11-3207d9f870a6-kube-api-access-zjdws\") pod \"payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b\" (UID: \"1913325b-12d7-4350-bb11-3207d9f870a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:30.387587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.387409    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab57aefb-f4f7-4061-94f7-8effed875817-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f\" (UID: \"ab57aefb-f4f7-4061-94f7-8effed875817\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:30.518488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.517552    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab57aefb-f4f7-4061-94f7-8effed875817-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f\" (UID: \"ab57aefb-f4f7-4061-94f7-8effed875817\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:30.518488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.517862    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxt7c\" (UniqueName: \"kubernetes.io/projected/ab57aefb-f4f7-4061-94f7-8effed875817-kube-api-access-qxt7c\") pod \"payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f\" (UID: \"ab57aefb-f4f7-4061-94f7-8effed875817\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:30.518488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.518018    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1913325b-12d7-4350-bb11-3207d9f870a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b\" (UID: \"1913325b-12d7-4350-bb11-3207d9f870a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:30.518488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.518120    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjdws\" (UniqueName: \"kubernetes.io/projected/1913325b-12d7-4350-bb11-3207d9f870a6-kube-api-access-zjdws\") pod \"payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b\" (UID: \"1913325b-12d7-4350-bb11-3207d9f870a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:30.533495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.519499    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab57aefb-f4f7-4061-94f7-8effed875817-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f\" (UID: \"ab57aefb-f4f7-4061-94f7-8effed875817\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:30.533495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.519686    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1913325b-12d7-4350-bb11-3207d9f870a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b\" (UID: \"1913325b-12d7-4350-bb11-3207d9f870a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:30.605549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.603254    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxt7c\" (UniqueName: \"kubernetes.io/projected/ab57aefb-f4f7-4061-94f7-8effed875817-kube-api-access-qxt7c\") pod \"payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f\" (UID: \"ab57aefb-f4f7-4061-94f7-8effed875817\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:30.605549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.603875    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjdws\" (UniqueName: \"kubernetes.io/projected/1913325b-12d7-4350-bb11-3207d9f870a6-kube-api-access-zjdws\") pod \"payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b\" (UID: \"1913325b-12d7-4350-bb11-3207d9f870a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:30.643492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.634715    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:30.653494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:30.648453    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:31.023995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:31.016239    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" event={"ID":"774525b8-c126-4535-a248-0b63d68e599a","Type":"ContainerStarted","Data":"3142b42eddc0ad8947f99d5f2f2c915dd010dab40d38d8237fb3efe073db150f"}
Oct 30 22:12:31.224233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:31.224202    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"]
Oct 30 22:12:31.225921 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:12:31.225892    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab57aefb_f4f7_4061_94f7_8effed875817.slice/crio-3c2f0a0e10a4aecd591eed5ca2f62bda2e5e27689f7db103cda57a671b33468c WatchSource:0}: Error finding container 3c2f0a0e10a4aecd591eed5ca2f62bda2e5e27689f7db103cda57a671b33468c: Status 404 returned error can't find the container with id 3c2f0a0e10a4aecd591eed5ca2f62bda2e5e27689f7db103cda57a671b33468c
Oct 30 22:12:31.229968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:31.229946    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"]
Oct 30 22:12:31.233400 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:12:31.233376    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1913325b_12d7_4350_bb11_3207d9f870a6.slice/crio-71cb535f88fff0f7654dd4d01cbfd0a2e71cf297c57c8dad4ecabbf656b20e4f WatchSource:0}: Error finding container 71cb535f88fff0f7654dd4d01cbfd0a2e71cf297c57c8dad4ecabbf656b20e4f: Status 404 returned error can't find the container with id 71cb535f88fff0f7654dd4d01cbfd0a2e71cf297c57c8dad4ecabbf656b20e4f
Oct 30 22:12:31.549778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:31.549698    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:12:31.551170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:31.551146    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:12:31.551289 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:31.551191    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:12:32.034164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.034125    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b" event={"ID":"1913325b-12d7-4350-bb11-3207d9f870a6","Type":"ContainerStarted","Data":"f11b8894c2685cbadadf803a4681389340c6051ecf122719c959ff5d74c975a8"}
Oct 30 22:12:32.035120 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.035077    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:32.035255 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.035130    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b" event={"ID":"1913325b-12d7-4350-bb11-3207d9f870a6","Type":"ContainerStarted","Data":"71cb535f88fff0f7654dd4d01cbfd0a2e71cf297c57c8dad4ecabbf656b20e4f"}
Oct 30 22:12:32.036344 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.036317    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f" event={"ID":"ab57aefb-f4f7-4061-94f7-8effed875817","Type":"ContainerStarted","Data":"b59f4a0648fd110d994375f9652bff75293594e6aa6fae23925ebf86e350e0ae"}
Oct 30 22:12:32.036492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.036371    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f" event={"ID":"ab57aefb-f4f7-4061-94f7-8effed875817","Type":"ContainerStarted","Data":"3c2f0a0e10a4aecd591eed5ca2f62bda2e5e27689f7db103cda57a671b33468c"}
Oct 30 22:12:32.077452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.077387    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b" podStartSLOduration=2.077369894 podStartE2EDuration="2.077369894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:32.050387801 +0000 UTC m=+1970.708484023" watchObservedRunningTime="2026-10-30 22:12:32.077369894 +0000 UTC m=+1970.735466117"
Oct 30 22:12:32.091545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.091491    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f" podStartSLOduration=2.091477396 podStartE2EDuration="2.091477396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:32.08683805 +0000 UTC m=+1970.744934273" watchObservedRunningTime="2026-10-30 22:12:32.091477396 +0000 UTC m=+1970.749573610"
Oct 30 22:12:32.482758 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.482722    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:12:32.552498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.550965    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:12:32.552498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:32.551037    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:12:33.044956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:33.044927    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:33.550401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:33.550369    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:12:33.550600 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:33.550420    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:12:34.550323 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:34.550291    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:12:34.550783 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:34.550346    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:12:34.749150 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:12:34.749116    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:12:35.549662 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:35.549621    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:12:35.549854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:35.549761    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:12:36.550483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:36.550444    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 22:12:36.550911 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:36.550541    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 22:12:37.554257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:37.554228    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:12:37.555573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:37.555551    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:12:43.048041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:43.048006    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:12:43.939869 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:43.939837    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"]
Oct 30 22:12:43.944890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:43.944865    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:43.956391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:43.956368    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6f8222-serving-cert\""
Oct 30 22:12:43.960323 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:43.960299    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"]
Oct 30 22:12:44.071708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.071671    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.072692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.071934    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-metrics-certs\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.072692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.071995    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cxn\" (UniqueName: \"kubernetes.io/projected/ed70ab40-99c6-412c-bf20-65031498ef71-kube-api-access-22cxn\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.072692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.072029    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed70ab40-99c6-412c-bf20-65031498ef71-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.080817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.080791    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:12:44.173719 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.173678    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-metrics-certs\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.173910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.173763    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cxn\" (UniqueName: \"kubernetes.io/projected/ed70ab40-99c6-412c-bf20-65031498ef71-kube-api-access-22cxn\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.173910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.173801    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed70ab40-99c6-412c-bf20-65031498ef71-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.174035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.174011    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.175789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.175764    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed70ab40-99c6-412c-bf20-65031498ef71-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.178119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.178094    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-metrics-certs\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.179334 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.179187    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.186422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.186403    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cxn\" (UniqueName: \"kubernetes.io/projected/ed70ab40-99c6-412c-bf20-65031498ef71-kube-api-access-22cxn\") pod \"maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.267182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.267105    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:44.642745 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.642723    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"]
Oct 30 22:12:44.642951 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:12:44.642804    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded70ab40_99c6_412c_bf20_65031498ef71.slice/crio-607696ba113cbbaedd775c09677330a67c52ad7defa4799724badd33d3a5dc0d WatchSource:0}: Error finding container 607696ba113cbbaedd775c09677330a67c52ad7defa4799724badd33d3a5dc0d: Status 404 returned error can't find the container with id 607696ba113cbbaedd775c09677330a67c52ad7defa4799724badd33d3a5dc0d
Oct 30 22:12:44.749132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:44.749098    4282 scope.go:122] "RemoveContainer" containerID="c9c89cbf0e5bcbe4bbfad871645e33ca46775784434cd827d56bc9790fe55858"
Oct 30 22:12:45.190541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:45.190504    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m" event={"ID":"ed70ab40-99c6-412c-bf20-65031498ef71","Type":"ContainerStarted","Data":"879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5"}
Oct 30 22:12:45.190951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:45.190561    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m" event={"ID":"ed70ab40-99c6-412c-bf20-65031498ef71","Type":"ContainerStarted","Data":"607696ba113cbbaedd775c09677330a67c52ad7defa4799724badd33d3a5dc0d"}
Oct 30 22:12:45.190951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:45.190597    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:12:45.213334 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:45.213278    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m" podStartSLOduration=1.7881946530000001 podStartE2EDuration="2.21326265s" totalImagesPullingTime="425.067997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:45.20871778 +0000 UTC m=+1983.866813994" watchObservedRunningTime="2026-10-30 22:12:45.21326265 +0000 UTC m=+1983.871358872"
Oct 30 22:12:51.204621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:12:51.204592    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:13:04.732525 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:13:04.730180    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:13:05.295257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:05.295225    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"]
Oct 30 22:13:07.870496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:07.866328    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:13:10.295987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.295936    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br" podUID="97fea9bb-c664-46d7-9d9f-224e228cf339" containerName="payload-processing" containerID="cri-o://0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd" gracePeriod=25
Oct 30 22:13:10.579610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.579578    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:13:10.642356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.642321    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-routing-overlay-volume\") pod \"97fea9bb-c664-46d7-9d9f-224e228cf339\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") "
Oct 30 22:13:10.642600 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.642578    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-plugins-config-volume\") pod \"97fea9bb-c664-46d7-9d9f-224e228cf339\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") "
Oct 30 22:13:10.642705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.642686    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-kube-api-access-j2k4f\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-kube-api-access-j2k4f\") pod \"97fea9bb-c664-46d7-9d9f-224e228cf339\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") "
Oct 30 22:13:10.642774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.642733    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-routing-overlay-volume" pod "97fea9bb-c664-46d7-9d9f-224e228cf339" (UID: "97fea9bb-c664-46d7-9d9f-224e228cf339"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:10.642774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.642766    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-provider-credentials\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-provider-credentials\") pod \"97fea9bb-c664-46d7-9d9f-224e228cf339\" (UID: \"97fea9bb-c664-46d7-9d9f-224e228cf339\") "
Oct 30 22:13:10.643359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.643329    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-plugins-config-volume" pod "97fea9bb-c664-46d7-9d9f-224e228cf339" (UID: "97fea9bb-c664-46d7-9d9f-224e228cf339"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:10.643701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.643622    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:10.643701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.643652    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fea9bb-c664-46d7-9d9f-224e228cf339-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:10.645433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.645410    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-provider-credentials" pod "97fea9bb-c664-46d7-9d9f-224e228cf339" (UID: "97fea9bb-c664-46d7-9d9f-224e228cf339"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:10.645768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.645743    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-kube-api-access-j2k4f" pod "97fea9bb-c664-46d7-9d9f-224e228cf339" (UID: "97fea9bb-c664-46d7-9d9f-224e228cf339"). InnerVolumeSpecName "kube-api-access-j2k4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:10.745409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.745376    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2k4f\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-kube-api-access-j2k4f\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:10.745409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.745405    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97fea9bb-c664-46d7-9d9f-224e228cf339-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:10.793394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.793358    4282 generic.go:356] "Generic (PLEG): container finished" podID="97fea9bb-c664-46d7-9d9f-224e228cf339" containerID="0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd" exitCode=0
Oct 30 22:13:10.793603 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.793523    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"
Oct 30 22:13:10.793686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.793521    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br" event={"ID":"97fea9bb-c664-46d7-9d9f-224e228cf339","Type":"ContainerDied","Data":"0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd"}
Oct 30 22:13:10.793748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.793688    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br" event={"ID":"97fea9bb-c664-46d7-9d9f-224e228cf339","Type":"ContainerDied","Data":"41046418ed29fcac2e997ce539dd02db6da3115644c080787d681513d39e71f7"}
Oct 30 22:13:10.793748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.793726    4282 scope.go:122] "RemoveContainer" containerID="0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd"
Oct 30 22:13:10.818509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.818457    4282 scope.go:122] "RemoveContainer" containerID="0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd"
Oct 30 22:13:10.818904 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:13:10.818877    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd\": container with ID starting with 0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd not found: ID does not exist" containerID="0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd"
Oct 30 22:13:10.819004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.818915    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd"} err="failed to get container status \"0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd\": rpc error: code = NotFound desc = could not find container \"0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd\": container with ID starting with 0a0e6b6950ee0eaa2e2f45caedbfe24b8bba78ac1a888bb855fa214351ffcfcd not found: ID does not exist"
Oct 30 22:13:10.837850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.837820    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"]
Oct 30 22:13:10.844720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.844692    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-59559n2br"]
Oct 30 22:13:10.920008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.919978    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"]
Oct 30 22:13:10.921572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.921271    4282 cpu_manager.go:414] "containerMap: removing container" podUID="97fea9bb-c664-46d7-9d9f-224e228cf339" containerName="payload-processing"
Oct 30 22:13:10.921572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.921293    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fea9bb-c664-46d7-9d9f-224e228cf339" containerName="payload-processing"
Oct 30 22:13:10.921572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.921532    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fea9bb-c664-46d7-9d9f-224e228cf339" containerName="payload-processing"
Oct 30 22:13:10.927007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.926988    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:10.939899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.939775    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"]
Oct 30 22:13:10.942887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.942853    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d8c835-dockercfg-hr9rq\""
Oct 30 22:13:10.943011 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.942945    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d8c835\""
Oct 30 22:13:10.944347 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.944326    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"]
Oct 30 22:13:10.948240 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.948209    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78322197-56a6-4b99-9e53-ab726a8fea53-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz\" (UID: \"78322197-56a6-4b99-9e53-ab726a8fea53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:10.950537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.948572    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdvt\" (UniqueName: \"kubernetes.io/projected/78322197-56a6-4b99-9e53-ab726a8fea53-kube-api-access-bkdvt\") pod \"payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz\" (UID: \"78322197-56a6-4b99-9e53-ab726a8fea53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:10.959647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.959618    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"]
Oct 30 22:13:10.959853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:10.959705    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:11.056852 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.056816    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c92a189-3606-4480-bb2d-d1a2281b4ea5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr\" (UID: \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:11.057284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.057261    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78322197-56a6-4b99-9e53-ab726a8fea53-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz\" (UID: \"78322197-56a6-4b99-9e53-ab726a8fea53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:11.058268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.058244    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj98w\" (UniqueName: \"kubernetes.io/projected/3c92a189-3606-4480-bb2d-d1a2281b4ea5-kube-api-access-hj98w\") pod \"payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr\" (UID: \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:11.058631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.058610    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdvt\" (UniqueName: \"kubernetes.io/projected/78322197-56a6-4b99-9e53-ab726a8fea53-kube-api-access-bkdvt\") pod \"payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz\" (UID: \"78322197-56a6-4b99-9e53-ab726a8fea53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:11.059444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.059424    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78322197-56a6-4b99-9e53-ab726a8fea53-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz\" (UID: \"78322197-56a6-4b99-9e53-ab726a8fea53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:11.114547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.110207    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdvt\" (UniqueName: \"kubernetes.io/projected/78322197-56a6-4b99-9e53-ab726a8fea53-kube-api-access-bkdvt\") pod \"payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz\" (UID: \"78322197-56a6-4b99-9e53-ab726a8fea53\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:11.168559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.167567    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj98w\" (UniqueName: \"kubernetes.io/projected/3c92a189-3606-4480-bb2d-d1a2281b4ea5-kube-api-access-hj98w\") pod \"payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr\" (UID: \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:11.168559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.167835    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c92a189-3606-4480-bb2d-d1a2281b4ea5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr\" (UID: \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:11.178965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.168862    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c92a189-3606-4480-bb2d-d1a2281b4ea5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr\" (UID: \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:11.197559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.194481    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj98w\" (UniqueName: \"kubernetes.io/projected/3c92a189-3606-4480-bb2d-d1a2281b4ea5-kube-api-access-hj98w\") pod \"payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr\" (UID: \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:11.257493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.256661    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:11.309492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.280223    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:11.482206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.482093    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fea9bb-c664-46d7-9d9f-224e228cf339" path="/var/lib/kubelet/pods/97fea9bb-c664-46d7-9d9f-224e228cf339/volumes"
Oct 30 22:13:11.665425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.665388    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"]
Oct 30 22:13:11.665685 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:13:11.665649    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78322197_56a6_4b99_9e53_ab726a8fea53.slice/crio-1bc5e10e8e1f7d8c3047edd94494d5795c0bf9c123816f3f530b88e8d6280d3f WatchSource:0}: Error finding container 1bc5e10e8e1f7d8c3047edd94494d5795c0bf9c123816f3f530b88e8d6280d3f: Status 404 returned error can't find the container with id 1bc5e10e8e1f7d8c3047edd94494d5795c0bf9c123816f3f530b88e8d6280d3f
Oct 30 22:13:11.678441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.677385    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"]
Oct 30 22:13:11.681160 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:13:11.681132    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c92a189_3606_4480_bb2d_d1a2281b4ea5.slice/crio-4ff774cdb491c9b2ca1164946e23e0e131bd43c7e5f3591d4910fdbb735811ab WatchSource:0}: Error finding container 4ff774cdb491c9b2ca1164946e23e0e131bd43c7e5f3591d4910fdbb735811ab: Status 404 returned error can't find the container with id 4ff774cdb491c9b2ca1164946e23e0e131bd43c7e5f3591d4910fdbb735811ab
Oct 30 22:13:11.937497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.934865    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr" event={"ID":"3c92a189-3606-4480-bb2d-d1a2281b4ea5","Type":"ContainerStarted","Data":"4ff774cdb491c9b2ca1164946e23e0e131bd43c7e5f3591d4910fdbb735811ab"}
Oct 30 22:13:11.952575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:11.949831    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz" event={"ID":"78322197-56a6-4b99-9e53-ab726a8fea53","Type":"ContainerStarted","Data":"1bc5e10e8e1f7d8c3047edd94494d5795c0bf9c123816f3f530b88e8d6280d3f"}
Oct 30 22:13:13.017505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:13.013657    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr" event={"ID":"3c92a189-3606-4480-bb2d-d1a2281b4ea5","Type":"ContainerStarted","Data":"440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1"}
Oct 30 22:13:13.017505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:13.015017    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:13.035626 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:13.031624    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz" event={"ID":"78322197-56a6-4b99-9e53-ab726a8fea53","Type":"ContainerStarted","Data":"e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61"}
Oct 30 22:13:13.035626 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:13.032660    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:13.093165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:13.093100    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr" podStartSLOduration=3.093075963 podStartE2EDuration="3.093075963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:13.070292518 +0000 UTC m=+2011.728388741" watchObservedRunningTime="2026-10-30 22:13:13.093075963 +0000 UTC m=+2011.751172186"
Oct 30 22:13:13.097245 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:13.097152    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz" podStartSLOduration=3.097140537 podStartE2EDuration="3.097140537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:13.089820169 +0000 UTC m=+2011.747916394" watchObservedRunningTime="2026-10-30 22:13:13.097140537 +0000 UTC m=+2011.755236761"
Oct 30 22:13:25.107007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:25.106788    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:13:25.125393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:25.125372    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:13:34.651966 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:13:34.651924    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:13:35.191939 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.191905    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"]
Oct 30 22:13:35.196951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.196920    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.206218 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.205839    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-28b4bb-openshift-default-dockercfg-bdj78\""
Oct 30 22:13:35.208616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.208595    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"]
Oct 30 22:13:35.315954 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.315920    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.316147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.315965    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhnp\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-kube-api-access-dlhnp\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.316147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.316123    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.316147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.316143    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.316329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.316234    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716883c7-6993-4ea2-b694-f334aba562c0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.316388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.316352    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.316547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.316531    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716883c7-6993-4ea2-b694-f334aba562c0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.316646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.316600    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.316705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.316642    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.417995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.417950    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.417995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.417998    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418066    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716883c7-6993-4ea2-b694-f334aba562c0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418129    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418247    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716883c7-6993-4ea2-b694-f334aba562c0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418421    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418455    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418507    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418427    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418659    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.418713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.418706    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhnp\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-kube-api-access-dlhnp\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.419085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.419027    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.419085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.419077    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716883c7-6993-4ea2-b694-f334aba562c0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.419252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.419077    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.421297 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.421265    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.421520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.421503    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716883c7-6993-4ea2-b694-f334aba562c0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.430240 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.430210    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.430701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.430680    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhnp\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-kube-api-access-dlhnp\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.517530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.517399    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:35.819019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.818986    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"]
Oct 30 22:13:35.823349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.823302    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:13:35.823498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.823403    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:13:35.823498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:35.823447    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:13:36.405233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:36.405194    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" event={"ID":"716883c7-6993-4ea2-b694-f334aba562c0","Type":"ContainerStarted","Data":"fcd8b766b5ce9cb275773b409311c243f507e96fe564d649043aa8dae887596f"}
Oct 30 22:13:36.405430 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:36.405263    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" event={"ID":"716883c7-6993-4ea2-b694-f334aba562c0","Type":"ContainerStarted","Data":"1261c177e63168266da69d486df8d5b624d1247101ecf2bace97921b17a6ac56"}
Oct 30 22:13:36.433865 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:36.433800    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" podStartSLOduration=1.4337820159999999 podStartE2EDuration="1.433782016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:36.426296728 +0000 UTC m=+2035.084392952" watchObservedRunningTime="2026-10-30 22:13:36.433782016 +0000 UTC m=+2035.091878240"
Oct 30 22:13:36.518402 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:36.518365    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:36.527367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:36.527333    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:13:36.527574 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:36.527406    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:13:37.518097 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:37.518063    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:13:37.518534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:37.518123    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:13:38.517768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:38.517736    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:13:38.517957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:38.517784    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:13:39.518432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:39.518397    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:13:39.518947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:39.518462    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:13:40.518662 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:40.518631    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:13:40.519182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:40.518722    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:13:41.517884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:41.517844    4282 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:13:41.518112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:41.517904    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 30 22:13:42.716494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:42.716128    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:42.720493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:42.717909    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:42.738492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:42.725622    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:13:47.906669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.906638    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"]
Oct 30 22:13:47.913021 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.912992    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:47.923099 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.923079    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-ead3143c31db\""
Oct 30 22:13:47.923249 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.923159    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-e2e--eb82ae76-dockercfg-zdlx9\""
Oct 30 22:13:47.925375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.925354    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"openshift-service-ca.crt\""
Oct 30 22:13:47.925540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.925362    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"routing-overlay\""
Oct 30 22:13:47.925686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.925525    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-28b4bb\""
Oct 30 22:13:47.925686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.925573    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"kube-root-ca.crt\""
Oct 30 22:13:47.925905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.925889    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"]
Oct 30 22:13:47.993990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.993948    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:47.994219 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.994016    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:47.994219 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.994134    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5j7w\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-kube-api-access-c5j7w\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:47.994313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:47.994266    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.020064 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.020031    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-813840a060cc\""
Oct 30 22:13:48.020213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.020120    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-3aae80dfab12\""
Oct 30 22:13:48.096518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.096453    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.096699 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.096550    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.096699 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.096616    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5j7w\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-kube-api-access-c5j7w\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.096850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.096748    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.097377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.097351    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.097517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.097426    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.099876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.099849    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.109297 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.109271    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5j7w\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-kube-api-access-c5j7w\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.230057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.230015    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.523624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.523597    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"]
Oct 30 22:13:48.524030 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:13:48.524001    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1407fc_0376_480d_bf3f_2e54baa58c01.slice/crio-a08f606bae394a4f99626ccd95bc1e9e23c0603e5aff64cbd8ae52b88d1bca20 WatchSource:0}: Error finding container a08f606bae394a4f99626ccd95bc1e9e23c0603e5aff64cbd8ae52b88d1bca20: Status 404 returned error can't find the container with id a08f606bae394a4f99626ccd95bc1e9e23c0603e5aff64cbd8ae52b88d1bca20
Oct 30 22:13:48.661806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.661778    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc" event={"ID":"da1407fc-0376-480d-bf3f-2e54baa58c01","Type":"ContainerStarted","Data":"817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18"}
Oct 30 22:13:48.661967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.661833    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc" event={"ID":"da1407fc-0376-480d-bf3f-2e54baa58c01","Type":"ContainerStarted","Data":"a08f606bae394a4f99626ccd95bc1e9e23c0603e5aff64cbd8ae52b88d1bca20"}
Oct 30 22:13:48.661967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.661896    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:48.677741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.677717    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"]
Oct 30 22:13:48.682537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.682517    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:48.687341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.687315    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"payload-processing-external-model-e2e--6ed667c1-dockercfg-6d7rx\""
Oct 30 22:13:48.689727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.689703    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"agc-cred-743a09bfc3d3\""
Oct 30 22:13:48.691578 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.691553    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"routing-overlay\""
Oct 30 22:13:48.692195 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.692157    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc" podStartSLOduration=1.692145107 podStartE2EDuration="1.692145107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:48.691289557 +0000 UTC m=+2047.349385781" watchObservedRunningTime="2026-10-30 22:13:48.692145107 +0000 UTC m=+2047.350241619"
Oct 30 22:13:48.692996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.692976    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"openshift-service-ca.crt\""
Oct 30 22:13:48.702543 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.702520    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"]
Oct 30 22:13:48.721163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.721128    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"kube-root-ca.crt\""
Oct 30 22:13:48.721302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.721222    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bacea0\""
Oct 30 22:13:48.864656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.864571    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:48.888491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.865127    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:48.888811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.888779    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbt9\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-kube-api-access-xnbt9\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:48.889080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.889059    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.001519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.998907    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.001519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.998966    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbt9\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-kube-api-access-xnbt9\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.001519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.999107    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.001519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:48.999319    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.001519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:49.000495    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.019494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:49.002897    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.025048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:49.024986    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.044906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:49.044843    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbt9\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-kube-api-access-xnbt9\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.298621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:49.298579    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:49.655380 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:49.655089    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"]
Oct 30 22:13:49.676251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:49.676216    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf" event={"ID":"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45","Type":"ContainerStarted","Data":"fe2edb3d3ce852b2d24f2195ec82e00150f93d52c8d870052442faf8112e0dd8"}
Oct 30 22:13:50.720008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:50.719963    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf" event={"ID":"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45","Type":"ContainerStarted","Data":"a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae"}
Oct 30 22:13:50.721091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:50.720362    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:50.750036 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:50.749984    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf" podStartSLOduration=2.74996779 podStartE2EDuration="2.74996779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:50.749721153 +0000 UTC m=+2049.407817402" watchObservedRunningTime="2026-10-30 22:13:50.74996779 +0000 UTC m=+2049.408064014"
Oct 30 22:13:54.522412 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:54.522383    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"]
Oct 30 22:13:54.525388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:54.525351    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:59.523390 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.523350    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf" podUID="8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" containerName="payload-processing" containerID="cri-o://a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae" gracePeriod=25
Oct 30 22:13:59.680107 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.680079    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:13:59.822209 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.822184    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:59.964581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.964550    4282 generic.go:356] "Generic (PLEG): container finished" podID="8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" containerID="a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae" exitCode=0
Oct 30 22:13:59.964788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.964606    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"
Oct 30 22:13:59.964788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.964674    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf" event={"ID":"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45","Type":"ContainerDied","Data":"a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae"}
Oct 30 22:13:59.964788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.964727    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf" event={"ID":"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45","Type":"ContainerDied","Data":"fe2edb3d3ce852b2d24f2195ec82e00150f93d52c8d870052442faf8112e0dd8"}
Oct 30 22:13:59.964788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.964750    4282 scope.go:122] "RemoveContainer" containerID="a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae"
Oct 30 22:13:59.996083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.996056    4282 scope.go:122] "RemoveContainer" containerID="a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae"
Oct 30 22:13:59.996516 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:13:59.996447    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae\": container with ID starting with a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae not found: ID does not exist" containerID="a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae"
Oct 30 22:13:59.996516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:13:59.996494    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae"} err="failed to get container status \"a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae\": rpc error: code = NotFound desc = could not find container \"a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae\": container with ID starting with a9104042c74cc02318b498c86d13f2cb856cc98e74d64994969389d34b6054ae not found: ID does not exist"
Oct 30 22:14:00.021353 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.021304    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-kube-api-access-xnbt9\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-kube-api-access-xnbt9\") pod \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") "
Oct 30 22:14:00.021631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.021556    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-provider-credentials\") pod \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") "
Oct 30 22:14:00.021767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.021752    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-plugins-config-volume\") pod \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") "
Oct 30 22:14:00.021891 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.021877    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-routing-overlay-volume\") pod \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\" (UID: \"8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45\") "
Oct 30 22:14:00.022220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.022191    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-plugins-config-volume" pod "8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" (UID: "8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:00.022321 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.022242    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-routing-overlay-volume" pod "8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" (UID: "8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:00.022603 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.022586    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:00.022649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.022607    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:00.023939 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.023913    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-provider-credentials" pod "8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" (UID: "8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:00.024038 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.023934    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-kube-api-access-xnbt9" pod "8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" (UID: "8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45"). InnerVolumeSpecName "kube-api-access-xnbt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:00.124139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.124099    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:00.124139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.124131    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbt9\" (UniqueName: \"kubernetes.io/projected/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45-kube-api-access-xnbt9\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:00.293528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.293406    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"]
Oct 30 22:14:00.301456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:00.301424    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-b77f84gnf"]
Oct 30 22:14:01.466526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:01.466484    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" path="/var/lib/kubelet/pods/8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45/volumes"
Oct 30 22:14:04.651318 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:14:04.651280    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:14:21.693342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.693306    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"]
Oct 30 22:14:21.694333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.694312    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" containerName="payload-processing"
Oct 30 22:14:21.694333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.694334    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" containerName="payload-processing"
Oct 30 22:14:21.694511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.694489    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbb4c3d-c4ad-409d-a61d-3fb6c2e53e45" containerName="payload-processing"
Oct 30 22:14:21.699364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.699346    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:21.703981 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.703961    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"payload-processing-external-model-e2e--96f1cc57-dockercfg-5bqlg\""
Oct 30 22:14:21.714414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.714391    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"]
Oct 30 22:14:21.716871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.716846    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"kube-root-ca.crt\""
Oct 30 22:14:21.716977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.716938    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"routing-overlay\""
Oct 30 22:14:21.717042 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.717021    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6f8222\""
Oct 30 22:14:21.717098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.717063    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"openshift-service-ca.crt\""
Oct 30 22:14:21.815868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.815839    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"agc-cred-17e7bc508a71\""
Oct 30 22:14:21.892591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.892556    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:21.893239 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.893201    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:21.893803 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.893781    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzzr8\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-kube-api-access-pzzr8\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:21.894031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.894014    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:21.995861 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.995773    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:21.997212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.997183    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzr8\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-kube-api-access-pzzr8\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:22.003438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:22.003413    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:22.007703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:22.007678    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:22.008694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:21.996937    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:22.008694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:22.007413    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:22.057572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:22.045453    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzzr8\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-kube-api-access-pzzr8\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:22.074488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:22.071260    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:22.316457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:22.316355    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:22.673530 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:14:22.673458    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a2c293_9e92_4573_a6c8_269466b6dc71.slice/crio-721626e9d756fb1ce78e05d279e646f9f272a6d13e4c47fa10e74c34def7c8cb WatchSource:0}: Error finding container 721626e9d756fb1ce78e05d279e646f9f272a6d13e4c47fa10e74c34def7c8cb: Status 404 returned error can't find the container with id 721626e9d756fb1ce78e05d279e646f9f272a6d13e4c47fa10e74c34def7c8cb
Oct 30 22:14:22.674254 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:22.674227    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"]
Oct 30 22:14:22.906677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:22.906641    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:14:23.372822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:23.372792    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw" event={"ID":"55a2c293-9e92-4573-a6c8-269466b6dc71","Type":"ContainerStarted","Data":"52d806444492a84cc466e1bdc4c611e5a745a3daa44d0169c92f7d3500fde61b"}
Oct 30 22:14:23.373037 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:23.372841    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw" event={"ID":"55a2c293-9e92-4573-a6c8-269466b6dc71","Type":"ContainerStarted","Data":"721626e9d756fb1ce78e05d279e646f9f272a6d13e4c47fa10e74c34def7c8cb"}
Oct 30 22:14:23.373037 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:23.372943    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:23.403067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:23.403014    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw" podStartSLOduration=2.4029999 podStartE2EDuration="2.4029999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:23.400695777 +0000 UTC m=+2082.058792002" watchObservedRunningTime="2026-10-30 22:14:23.4029999 +0000 UTC m=+2082.061096122"
Oct 30 22:14:34.408031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:34.408003    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:14:34.662877 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:14:34.662786    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:14:39.914232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:39.914193    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"]
Oct 30 22:14:44.914663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:44.914615    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc" podUID="da1407fc-0376-480d-bf3f-2e54baa58c01" containerName="payload-processing" containerID="cri-o://817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18" gracePeriod=25
Oct 30 22:14:45.224764 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.224735    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:14:45.280414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.280369    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-plugins-config-volume\") pod \"da1407fc-0376-480d-bf3f-2e54baa58c01\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") "
Oct 30 22:14:45.280648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.280527    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-provider-credentials\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-provider-credentials\") pod \"da1407fc-0376-480d-bf3f-2e54baa58c01\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") "
Oct 30 22:14:45.280648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.280642    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-kube-api-access-c5j7w\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-kube-api-access-c5j7w\") pod \"da1407fc-0376-480d-bf3f-2e54baa58c01\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") "
Oct 30 22:14:45.280881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.280833    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-plugins-config-volume" pod "da1407fc-0376-480d-bf3f-2e54baa58c01" (UID: "da1407fc-0376-480d-bf3f-2e54baa58c01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:45.281130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.281108    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-routing-overlay-volume\") pod \"da1407fc-0376-480d-bf3f-2e54baa58c01\" (UID: \"da1407fc-0376-480d-bf3f-2e54baa58c01\") "
Oct 30 22:14:45.281535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.281510    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-routing-overlay-volume" pod "da1407fc-0376-480d-bf3f-2e54baa58c01" (UID: "da1407fc-0376-480d-bf3f-2e54baa58c01"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:45.282302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.282281    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:45.282398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.282304    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da1407fc-0376-480d-bf3f-2e54baa58c01-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:45.283998 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.283973    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-provider-credentials" pod "da1407fc-0376-480d-bf3f-2e54baa58c01" (UID: "da1407fc-0376-480d-bf3f-2e54baa58c01"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:45.284116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.284011    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-kube-api-access-c5j7w" pod "da1407fc-0376-480d-bf3f-2e54baa58c01" (UID: "da1407fc-0376-480d-bf3f-2e54baa58c01"). InnerVolumeSpecName "kube-api-access-c5j7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:45.383925 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.383895    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:45.383925 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.383923    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5j7w\" (UniqueName: \"kubernetes.io/projected/da1407fc-0376-480d-bf3f-2e54baa58c01-kube-api-access-c5j7w\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:45.910841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.910758    4282 generic.go:356] "Generic (PLEG): container finished" podID="da1407fc-0376-480d-bf3f-2e54baa58c01" containerID="817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18" exitCode=0
Oct 30 22:14:45.910841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.910805    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"
Oct 30 22:14:45.911035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.910839    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc" event={"ID":"da1407fc-0376-480d-bf3f-2e54baa58c01","Type":"ContainerDied","Data":"817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18"}
Oct 30 22:14:45.911035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.910890    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc" event={"ID":"da1407fc-0376-480d-bf3f-2e54baa58c01","Type":"ContainerDied","Data":"a08f606bae394a4f99626ccd95bc1e9e23c0603e5aff64cbd8ae52b88d1bca20"}
Oct 30 22:14:45.911035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.910912    4282 scope.go:122] "RemoveContainer" containerID="817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18"
Oct 30 22:14:45.924932 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.924908    4282 scope.go:122] "RemoveContainer" containerID="817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18"
Oct 30 22:14:45.925831 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:14:45.925587    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18\": container with ID starting with 817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18 not found: ID does not exist" containerID="817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18"
Oct 30 22:14:45.925831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.925628    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18"} err="failed to get container status \"817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18\": rpc error: code = NotFound desc = could not find container \"817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18\": container with ID starting with 817092afeede89a8c5717241451f43959422980dfa81c49dbb5af2aecaa1fd18 not found: ID does not exist"
Oct 30 22:14:45.936479 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.936445    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"]
Oct 30 22:14:45.943727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:45.943705    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-84d9x4slc"]
Oct 30 22:14:47.466894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:47.466860    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1407fc-0376-480d-bf3f-2e54baa58c01" path="/var/lib/kubelet/pods/da1407fc-0376-480d-bf3f-2e54baa58c01/volumes"
Oct 30 22:14:48.818495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.815113    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"]
Oct 30 22:14:48.818495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.816757    4282 cpu_manager.go:414] "containerMap: removing container" podUID="da1407fc-0376-480d-bf3f-2e54baa58c01" containerName="payload-processing"
Oct 30 22:14:48.818495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.816781    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1407fc-0376-480d-bf3f-2e54baa58c01" containerName="payload-processing"
Oct 30 22:14:48.818495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.817051    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1407fc-0376-480d-bf3f-2e54baa58c01" containerName="payload-processing"
Oct 30 22:14:48.835238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.835088    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:48.847500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.839413    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"]
Oct 30 22:14:48.847500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.844609    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"routing-overlay\""
Oct 30 22:14:48.847500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.844683    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"payload-processing-external-model-e2e--a9381736-dockercfg-kqh8l\""
Oct 30 22:14:48.847500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.844737    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"agc-cred-3261044a2004\""
Oct 30 22:14:48.847500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.844795    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"payload-processing-external-model-plugins-e2e-extmod-b-26a128\""
Oct 30 22:14:48.847500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.844839    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"openshift-service-ca.crt\""
Oct 30 22:14:48.847500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.844894    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-26a128\"/\"kube-root-ca.crt\""
Oct 30 22:14:48.972493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.971895    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:48.972493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.972054    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:48.972493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.972108    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdzxw\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-kube-api-access-gdzxw\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:48.972493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:48.972159    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.089007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.087979    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.089007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.088077    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzxw\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-kube-api-access-gdzxw\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.089007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.088151    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.089007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.088345    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.090523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.090074    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.091037 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.090986    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.104740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.104674    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.112745 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.111313    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdzxw\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-kube-api-access-gdzxw\") pod \"payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") " pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.208932 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.208894    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:49.497880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:49.497849    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"]
Oct 30 22:14:49.498488 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:14:49.498442    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ad18a4_c94a_455b_b097_ad268e10f623.slice/crio-556cea034f9c1495d5812e276af32cfe18a0319ab15369d797f5849eac827f97 WatchSource:0}: Error finding container 556cea034f9c1495d5812e276af32cfe18a0319ab15369d797f5849eac827f97: Status 404 returned error can't find the container with id 556cea034f9c1495d5812e276af32cfe18a0319ab15369d797f5849eac827f97
Oct 30 22:14:50.107480 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:50.107439    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh" event={"ID":"23ad18a4-c94a-455b-b097-ad268e10f623","Type":"ContainerStarted","Data":"80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176"}
Oct 30 22:14:50.107932 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:50.107515    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh" event={"ID":"23ad18a4-c94a-455b-b097-ad268e10f623","Type":"ContainerStarted","Data":"556cea034f9c1495d5812e276af32cfe18a0319ab15369d797f5849eac827f97"}
Oct 30 22:14:50.107932 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:50.107707    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:14:50.165890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:14:50.165837    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh" podStartSLOduration=2.165821145 podStartE2EDuration="2.165821145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:50.161063644 +0000 UTC m=+2108.819159877" watchObservedRunningTime="2026-10-30 22:14:50.165821145 +0000 UTC m=+2108.823917368"
Oct 30 22:15:00.310539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.310306    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"]
Oct 30 22:15:00.327489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.327445    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:00.333692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.331922    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"]
Oct 30 22:15:00.346738 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.346709    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\""
Oct 30 22:15:00.347790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.347765    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"]
Oct 30 22:15:00.347986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.347867    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:00.369019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.368986    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"]
Oct 30 22:15:00.376151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.376120    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"]
Oct 30 22:15:00.381669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.381644    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:00.408568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.406430    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"]
Oct 30 22:15:00.411686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.411653    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0991b6-ebfa-4f94-b400-df053996b279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm\" (UID: \"ec0991b6-ebfa-4f94-b400-df053996b279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:00.412077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.412048    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxvj\" (UniqueName: \"kubernetes.io/projected/ec0991b6-ebfa-4f94-b400-df053996b279-kube-api-access-qvxvj\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm\" (UID: \"ec0991b6-ebfa-4f94-b400-df053996b279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:00.412131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.412100    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b44c3-a9f0-4010-9ac0-5a4f42335537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh\" (UID: \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:00.412176 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.412159    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcxr\" (UniqueName: \"kubernetes.io/projected/f70b44c3-a9f0-4010-9ac0-5a4f42335537-kube-api-access-qzcxr\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh\" (UID: \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:00.414187 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.414161    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"]
Oct 30 22:15:00.414314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.414292    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:00.451275 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.451059    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"]
Oct 30 22:15:00.465684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.465645    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"]
Oct 30 22:15:00.471367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.471334    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:00.471845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.471787    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"]
Oct 30 22:15:00.480861 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.479519    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"]
Oct 30 22:15:00.492830 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.492773    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:00.499577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.499089    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"]
Oct 30 22:15:00.514287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.513591    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxvj\" (UniqueName: \"kubernetes.io/projected/ec0991b6-ebfa-4f94-b400-df053996b279-kube-api-access-qvxvj\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm\" (UID: \"ec0991b6-ebfa-4f94-b400-df053996b279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:00.514287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.513631    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b44c3-a9f0-4010-9ac0-5a4f42335537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh\" (UID: \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:00.514287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.513686    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b3fefe-386a-4d09-aad5-857c41bad582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c\" (UID: \"a3b3fefe-386a-4d09-aad5-857c41bad582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:00.514287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.513714    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcxr\" (UniqueName: \"kubernetes.io/projected/f70b44c3-a9f0-4010-9ac0-5a4f42335537-kube-api-access-qzcxr\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh\" (UID: \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:00.514287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.513770    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94837e5c-a04e-47d7-8b11-85b4ec07cb87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv\" (UID: \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:00.514287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.513824    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gwz5\" (UniqueName: \"kubernetes.io/projected/a3b3fefe-386a-4d09-aad5-857c41bad582-kube-api-access-8gwz5\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c\" (UID: \"a3b3fefe-386a-4d09-aad5-857c41bad582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:00.514287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.513975    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmw2\" (UniqueName: \"kubernetes.io/projected/94837e5c-a04e-47d7-8b11-85b4ec07cb87-kube-api-access-5zmw2\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv\" (UID: \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:00.514287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.514144    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0991b6-ebfa-4f94-b400-df053996b279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm\" (UID: \"ec0991b6-ebfa-4f94-b400-df053996b279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:00.515004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.514798    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b44c3-a9f0-4010-9ac0-5a4f42335537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh\" (UID: \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:00.515517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.515490    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0991b6-ebfa-4f94-b400-df053996b279-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm\" (UID: \"ec0991b6-ebfa-4f94-b400-df053996b279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:00.544143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.544105    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxvj\" (UniqueName: \"kubernetes.io/projected/ec0991b6-ebfa-4f94-b400-df053996b279-kube-api-access-qvxvj\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm\" (UID: \"ec0991b6-ebfa-4f94-b400-df053996b279\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:00.544324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.544227    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcxr\" (UniqueName: \"kubernetes.io/projected/f70b44c3-a9f0-4010-9ac0-5a4f42335537-kube-api-access-qzcxr\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh\" (UID: \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:00.615902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.615815    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b3fefe-386a-4d09-aad5-857c41bad582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c\" (UID: \"a3b3fefe-386a-4d09-aad5-857c41bad582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:00.615902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.615880    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94837e5c-a04e-47d7-8b11-85b4ec07cb87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv\" (UID: \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:00.616183 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.615928    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zh5n\" (UniqueName: \"kubernetes.io/projected/02609366-36d2-492b-9aed-c0afc70b9b6e-kube-api-access-5zh5n\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs\" (UID: \"02609366-36d2-492b-9aed-c0afc70b9b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:00.616183 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.615953    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gwz5\" (UniqueName: \"kubernetes.io/projected/a3b3fefe-386a-4d09-aad5-857c41bad582-kube-api-access-8gwz5\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c\" (UID: \"a3b3fefe-386a-4d09-aad5-857c41bad582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:00.616307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.616198    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmw2\" (UniqueName: \"kubernetes.io/projected/94837e5c-a04e-47d7-8b11-85b4ec07cb87-kube-api-access-5zmw2\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv\" (UID: \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:00.616307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.616280    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4wxf\" (UniqueName: \"kubernetes.io/projected/65dc0398-a945-4341-893b-53cca23a17ea-kube-api-access-f4wxf\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf\" (UID: \"65dc0398-a945-4341-893b-53cca23a17ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:00.616489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.616446    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc0398-a945-4341-893b-53cca23a17ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf\" (UID: \"65dc0398-a945-4341-893b-53cca23a17ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:00.616588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.616570    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02609366-36d2-492b-9aed-c0afc70b9b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs\" (UID: \"02609366-36d2-492b-9aed-c0afc70b9b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:00.616663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.616627    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b3fefe-386a-4d09-aad5-857c41bad582-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c\" (UID: \"a3b3fefe-386a-4d09-aad5-857c41bad582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:00.616797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.616775    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94837e5c-a04e-47d7-8b11-85b4ec07cb87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv\" (UID: \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:00.634444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.634406    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gwz5\" (UniqueName: \"kubernetes.io/projected/a3b3fefe-386a-4d09-aad5-857c41bad582-kube-api-access-8gwz5\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c\" (UID: \"a3b3fefe-386a-4d09-aad5-857c41bad582\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:00.641311 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.641286    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmw2\" (UniqueName: \"kubernetes.io/projected/94837e5c-a04e-47d7-8b11-85b4ec07cb87-kube-api-access-5zmw2\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv\" (UID: \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:00.656865 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.656841    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:00.672337 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.672313    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:00.702442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.702408    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:00.718692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.718662    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc0398-a945-4341-893b-53cca23a17ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf\" (UID: \"65dc0398-a945-4341-893b-53cca23a17ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:00.718853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.718709    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02609366-36d2-492b-9aed-c0afc70b9b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs\" (UID: \"02609366-36d2-492b-9aed-c0afc70b9b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:00.718853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.718836    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zh5n\" (UniqueName: \"kubernetes.io/projected/02609366-36d2-492b-9aed-c0afc70b9b6e-kube-api-access-5zh5n\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs\" (UID: \"02609366-36d2-492b-9aed-c0afc70b9b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:00.719049 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.718973    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4wxf\" (UniqueName: \"kubernetes.io/projected/65dc0398-a945-4341-893b-53cca23a17ea-kube-api-access-f4wxf\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf\" (UID: \"65dc0398-a945-4341-893b-53cca23a17ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:00.720854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.720818    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02609366-36d2-492b-9aed-c0afc70b9b6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs\" (UID: \"02609366-36d2-492b-9aed-c0afc70b9b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:00.726074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.722111    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc0398-a945-4341-893b-53cca23a17ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf\" (UID: \"65dc0398-a945-4341-893b-53cca23a17ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:00.737920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.737850    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zh5n\" (UniqueName: \"kubernetes.io/projected/02609366-36d2-492b-9aed-c0afc70b9b6e-kube-api-access-5zh5n\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs\" (UID: \"02609366-36d2-492b-9aed-c0afc70b9b6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:00.739071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.739040    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4wxf\" (UniqueName: \"kubernetes.io/projected/65dc0398-a945-4341-893b-53cca23a17ea-kube-api-access-f4wxf\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf\" (UID: \"65dc0398-a945-4341-893b-53cca23a17ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:00.752032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.751792    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:00.798780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.798749    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:00.813385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.813350    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:00.968380 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:00.967267    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"]
Oct 30 22:15:00.973774 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:00.973744    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0991b6_ebfa_4f94_b400_df053996b279.slice/crio-ecbc157bae893af690b2184b1b553ad33d915aea2dcb0d513f8d7870e5b3173c WatchSource:0}: Error finding container ecbc157bae893af690b2184b1b553ad33d915aea2dcb0d513f8d7870e5b3173c: Status 404 returned error can't find the container with id ecbc157bae893af690b2184b1b553ad33d915aea2dcb0d513f8d7870e5b3173c
Oct 30 22:15:01.118782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.118701    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:15:01.122313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.122281    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"]
Oct 30 22:15:01.123409 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:01.123386    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b3fefe_386a_4d09_aad5_857c41bad582.slice/crio-d2933a7d8eacc28c22bdcd5b4091a09b66ad99d9e18159b5e321b3da30268a11 WatchSource:0}: Error finding container d2933a7d8eacc28c22bdcd5b4091a09b66ad99d9e18159b5e321b3da30268a11: Status 404 returned error can't find the container with id d2933a7d8eacc28c22bdcd5b4091a09b66ad99d9e18159b5e321b3da30268a11
Oct 30 22:15:01.158845 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:01.158166    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02609366_36d2_492b_9aed_c0afc70b9b6e.slice/crio-4e1b8142ad9c843f8941a45420c37f3ba064303065e9dce64f48221750298cd7 WatchSource:0}: Error finding container 4e1b8142ad9c843f8941a45420c37f3ba064303065e9dce64f48221750298cd7: Status 404 returned error can't find the container with id 4e1b8142ad9c843f8941a45420c37f3ba064303065e9dce64f48221750298cd7
Oct 30 22:15:01.160680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.160656    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"]
Oct 30 22:15:01.172559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.172147    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"]
Oct 30 22:15:01.180047 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:01.180016    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dc0398_a945_4341_893b_53cca23a17ea.slice/crio-d5536268ad4e4cf333c85bac14c72daaa961ac61c5ed75ab814559c2b4b3efe3 WatchSource:0}: Error finding container d5536268ad4e4cf333c85bac14c72daaa961ac61c5ed75ab814559c2b4b3efe3: Status 404 returned error can't find the container with id d5536268ad4e4cf333c85bac14c72daaa961ac61c5ed75ab814559c2b4b3efe3
Oct 30 22:15:01.223104 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.223073    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"]
Oct 30 22:15:01.228991 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:01.228967    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70b44c3_a9f0_4010_9ac0_5a4f42335537.slice/crio-ef459bda33e8fc34cf8f387369d98460ee0dd39cb4c36284e64c08dfe056517d WatchSource:0}: Error finding container ef459bda33e8fc34cf8f387369d98460ee0dd39cb4c36284e64c08dfe056517d: Status 404 returned error can't find the container with id ef459bda33e8fc34cf8f387369d98460ee0dd39cb4c36284e64c08dfe056517d
Oct 30 22:15:01.264274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.264238    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" event={"ID":"a3b3fefe-386a-4d09-aad5-857c41bad582","Type":"ContainerStarted","Data":"d2933a7d8eacc28c22bdcd5b4091a09b66ad99d9e18159b5e321b3da30268a11"}
Oct 30 22:15:01.266436 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.266406    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" event={"ID":"ec0991b6-ebfa-4f94-b400-df053996b279","Type":"ContainerStarted","Data":"ecbc157bae893af690b2184b1b553ad33d915aea2dcb0d513f8d7870e5b3173c"}
Oct 30 22:15:01.268374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.268346    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" event={"ID":"02609366-36d2-492b-9aed-c0afc70b9b6e","Type":"ContainerStarted","Data":"4e1b8142ad9c843f8941a45420c37f3ba064303065e9dce64f48221750298cd7"}
Oct 30 22:15:01.275963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.275934    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" event={"ID":"65dc0398-a945-4341-893b-53cca23a17ea","Type":"ContainerStarted","Data":"d5536268ad4e4cf333c85bac14c72daaa961ac61c5ed75ab814559c2b4b3efe3"}
Oct 30 22:15:01.279212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.279180    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" event={"ID":"f70b44c3-a9f0-4010-9ac0-5a4f42335537","Type":"ContainerStarted","Data":"ef459bda33e8fc34cf8f387369d98460ee0dd39cb4c36284e64c08dfe056517d"}
Oct 30 22:15:01.280086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:01.280061    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"]
Oct 30 22:15:01.440970 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:01.440929    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94837e5c_a04e_47d7_8b11_85b4ec07cb87.slice/crio-720e425b832dcb91eba8a408db32c4fd87f9b4039a8f692cc2476e29e14af6e8 WatchSource:0}: Error finding container 720e425b832dcb91eba8a408db32c4fd87f9b4039a8f692cc2476e29e14af6e8: Status 404 returned error can't find the container with id 720e425b832dcb91eba8a408db32c4fd87f9b4039a8f692cc2476e29e14af6e8
Oct 30 22:15:02.686583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:02.649909    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" event={"ID":"65dc0398-a945-4341-893b-53cca23a17ea","Type":"ContainerStarted","Data":"2c5a41b1ce9d441fcdd22780871da88860565ee2496fa768d74d5ab90fc80fc9"}
Oct 30 22:15:02.747999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:02.741526    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" podStartSLOduration=2.7415083879999997 podStartE2EDuration="2.741508388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.740014838 +0000 UTC m=+2121.398111062" watchObservedRunningTime="2026-10-30 22:15:02.741508388 +0000 UTC m=+2121.399604608"
Oct 30 22:15:02.986121 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:02.986002    4282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02609366_36d2_492b_9aed_c0afc70b9b6e.slice/crio-conmon-6b15f9b397eec19622a6fc3fb884298725e07762a565c08ed6e6518e2660dc18.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0991b6_ebfa_4f94_b400_df053996b279.slice/crio-cfd594ffaa8ab216b7d88ac0b73e65b6f6f7b729dd081721eec0673d9c67f4ef.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dc0398_a945_4341_893b_53cca23a17ea.slice/crio-2c5a41b1ce9d441fcdd22780871da88860565ee2496fa768d74d5ab90fc80fc9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0991b6_ebfa_4f94_b400_df053996b279.slice/crio-conmon-cfd594ffaa8ab216b7d88ac0b73e65b6f6f7b729dd081721eec0673d9c67f4ef.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94837e5c_a04e_47d7_8b11_85b4ec07cb87.slice/crio-conmon-8e44499325a85fc815ce62e0d35e7d274ec94bef2274094d145083951f81da08.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b3fefe_386a_4d09_aad5_857c41bad582.slice/crio-57627f8c45a1c8f64e255acfc2935aac7e84d7bcf80b68e4cb634ea43e4689b8.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:15:02.996182 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:02.995858    4282 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02609366_36d2_492b_9aed_c0afc70b9b6e.slice/crio-6b15f9b397eec19622a6fc3fb884298725e07762a565c08ed6e6518e2660dc18.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94837e5c_a04e_47d7_8b11_85b4ec07cb87.slice/crio-8e44499325a85fc815ce62e0d35e7d274ec94bef2274094d145083951f81da08.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b3fefe_386a_4d09_aad5_857c41bad582.slice/crio-conmon-57627f8c45a1c8f64e255acfc2935aac7e84d7bcf80b68e4cb634ea43e4689b8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02609366_36d2_492b_9aed_c0afc70b9b6e.slice/crio-conmon-6b15f9b397eec19622a6fc3fb884298725e07762a565c08ed6e6518e2660dc18.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:15:03.084576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.084522    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" event={"ID":"f70b44c3-a9f0-4010-9ac0-5a4f42335537","Type":"ContainerStarted","Data":"f1ff99233eec06f93377513d4e698870f977d262ef36c604ff8d7ee8325d432e"}
Oct 30 22:15:03.091140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.091033    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" event={"ID":"94837e5c-a04e-47d7-8b11-85b4ec07cb87","Type":"ContainerStarted","Data":"8e44499325a85fc815ce62e0d35e7d274ec94bef2274094d145083951f81da08"}
Oct 30 22:15:03.091379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.091359    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" event={"ID":"94837e5c-a04e-47d7-8b11-85b4ec07cb87","Type":"ContainerStarted","Data":"720e425b832dcb91eba8a408db32c4fd87f9b4039a8f692cc2476e29e14af6e8"}
Oct 30 22:15:03.098207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.098168    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" event={"ID":"a3b3fefe-386a-4d09-aad5-857c41bad582","Type":"ContainerStarted","Data":"57627f8c45a1c8f64e255acfc2935aac7e84d7bcf80b68e4cb634ea43e4689b8"}
Oct 30 22:15:03.133507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.125243    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" event={"ID":"ec0991b6-ebfa-4f94-b400-df053996b279","Type":"ContainerStarted","Data":"cfd594ffaa8ab216b7d88ac0b73e65b6f6f7b729dd081721eec0673d9c67f4ef"}
Oct 30 22:15:03.133507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.125821    4282 scope.go:122] "RemoveContainer" containerID="cfd594ffaa8ab216b7d88ac0b73e65b6f6f7b729dd081721eec0673d9c67f4ef"
Oct 30 22:15:03.134247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.134186    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" podStartSLOduration=3.134165001 podStartE2EDuration="3.134165001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.123728563 +0000 UTC m=+2121.781824787" watchObservedRunningTime="2026-10-30 22:15:03.134165001 +0000 UTC m=+2121.792261225"
Oct 30 22:15:03.211512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.211455    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" event={"ID":"02609366-36d2-492b-9aed-c0afc70b9b6e","Type":"ContainerStarted","Data":"6b15f9b397eec19622a6fc3fb884298725e07762a565c08ed6e6518e2660dc18"}
Oct 30 22:15:03.246809 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.246708    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" podStartSLOduration=3.246687181 podStartE2EDuration="3.246687181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.205526074 +0000 UTC m=+2121.863622298" watchObservedRunningTime="2026-10-30 22:15:03.246687181 +0000 UTC m=+2121.904783410"
Oct 30 22:15:03.269786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.269726    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" podStartSLOduration=3.269706182 podStartE2EDuration="3.269706182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.264527658 +0000 UTC m=+2121.922623882" watchObservedRunningTime="2026-10-30 22:15:03.269706182 +0000 UTC m=+2121.927802406"
Oct 30 22:15:03.325777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:03.325716    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" podStartSLOduration=3.3257003 podStartE2EDuration="3.3257003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.325031884 +0000 UTC m=+2121.983128108" watchObservedRunningTime="2026-10-30 22:15:03.3257003 +0000 UTC m=+2121.983796525"
Oct 30 22:15:04.256166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.254443    4282 generic.go:356] "Generic (PLEG): container finished" podID="65dc0398-a945-4341-893b-53cca23a17ea" containerID="2c5a41b1ce9d441fcdd22780871da88860565ee2496fa768d74d5ab90fc80fc9" exitCode=60
Oct 30 22:15:04.260494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.256797    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" event={"ID":"65dc0398-a945-4341-893b-53cca23a17ea","Type":"ContainerDied","Data":"2c5a41b1ce9d441fcdd22780871da88860565ee2496fa768d74d5ab90fc80fc9"}
Oct 30 22:15:04.261215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.261190    4282 scope.go:122] "RemoveContainer" containerID="2c5a41b1ce9d441fcdd22780871da88860565ee2496fa768d74d5ab90fc80fc9"
Oct 30 22:15:04.428512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.425813    4282 generic.go:356] "Generic (PLEG): container finished" podID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerID="f1ff99233eec06f93377513d4e698870f977d262ef36c604ff8d7ee8325d432e" exitCode=60
Oct 30 22:15:04.428512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.426077    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" event={"ID":"f70b44c3-a9f0-4010-9ac0-5a4f42335537","Type":"ContainerDied","Data":"f1ff99233eec06f93377513d4e698870f977d262ef36c604ff8d7ee8325d432e"}
Oct 30 22:15:04.428512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.426740    4282 scope.go:122] "RemoveContainer" containerID="f1ff99233eec06f93377513d4e698870f977d262ef36c604ff8d7ee8325d432e"
Oct 30 22:15:04.487314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.487282    4282 generic.go:356] "Generic (PLEG): container finished" podID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerID="8e44499325a85fc815ce62e0d35e7d274ec94bef2274094d145083951f81da08" exitCode=60
Oct 30 22:15:04.487676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.487651    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" event={"ID":"94837e5c-a04e-47d7-8b11-85b4ec07cb87","Type":"ContainerDied","Data":"8e44499325a85fc815ce62e0d35e7d274ec94bef2274094d145083951f81da08"}
Oct 30 22:15:04.490227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.490202    4282 generic.go:356] "Generic (PLEG): container finished" podID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerID="57627f8c45a1c8f64e255acfc2935aac7e84d7bcf80b68e4cb634ea43e4689b8" exitCode=60
Oct 30 22:15:04.490489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.490453    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" event={"ID":"a3b3fefe-386a-4d09-aad5-857c41bad582","Type":"ContainerDied","Data":"57627f8c45a1c8f64e255acfc2935aac7e84d7bcf80b68e4cb634ea43e4689b8"}
Oct 30 22:15:04.494518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.494493    4282 generic.go:356] "Generic (PLEG): container finished" podID="ec0991b6-ebfa-4f94-b400-df053996b279" containerID="cfd594ffaa8ab216b7d88ac0b73e65b6f6f7b729dd081721eec0673d9c67f4ef" exitCode=60
Oct 30 22:15:04.494735 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.494717    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" event={"ID":"ec0991b6-ebfa-4f94-b400-df053996b279","Type":"ContainerDied","Data":"cfd594ffaa8ab216b7d88ac0b73e65b6f6f7b729dd081721eec0673d9c67f4ef"}
Oct 30 22:15:04.494863 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.494849    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" event={"ID":"ec0991b6-ebfa-4f94-b400-df053996b279","Type":"ContainerStarted","Data":"720ab40bfa1297850d6e5b5039c3f751d36e2c76b8a150a5fa2936eee77f2ce7"}
Oct 30 22:15:04.496909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.496889    4282 generic.go:356] "Generic (PLEG): container finished" podID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerID="6b15f9b397eec19622a6fc3fb884298725e07762a565c08ed6e6518e2660dc18" exitCode=60
Oct 30 22:15:04.497260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.497246    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" event={"ID":"02609366-36d2-492b-9aed-c0afc70b9b6e","Type":"ContainerDied","Data":"6b15f9b397eec19622a6fc3fb884298725e07762a565c08ed6e6518e2660dc18"}
Oct 30 22:15:04.502715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.502694    4282 scope.go:122] "RemoveContainer" containerID="8e44499325a85fc815ce62e0d35e7d274ec94bef2274094d145083951f81da08"
Oct 30 22:15:04.503742 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.503723    4282 scope.go:122] "RemoveContainer" containerID="57627f8c45a1c8f64e255acfc2935aac7e84d7bcf80b68e4cb634ea43e4689b8"
Oct 30 22:15:04.512153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.512091    4282 scope.go:122] "RemoveContainer" containerID="6b15f9b397eec19622a6fc3fb884298725e07762a565c08ed6e6518e2660dc18"
Oct 30 22:15:04.552507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:04.552357    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" podStartSLOduration=4.552338182 podStartE2EDuration="4.552338182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.544780221 +0000 UTC m=+2123.202876444" watchObservedRunningTime="2026-10-30 22:15:04.552338182 +0000 UTC m=+2123.210434410"
Oct 30 22:15:04.982640 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:04.982609    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:15:05.507186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.507151    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" event={"ID":"94837e5c-a04e-47d7-8b11-85b4ec07cb87","Type":"ContainerStarted","Data":"1e315879552cf56560f2dff46ca108d2918f719578764463a16a9d8b0a4ff183"}
Oct 30 22:15:05.509558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.509528    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" event={"ID":"a3b3fefe-386a-4d09-aad5-857c41bad582","Type":"ContainerStarted","Data":"37ea822ca2dd639864dbc3212b513b934dc820d8ca4d2353f378a6721da8557d"}
Oct 30 22:15:05.511920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.511892    4282 generic.go:356] "Generic (PLEG): container finished" podID="ec0991b6-ebfa-4f94-b400-df053996b279" containerID="720ab40bfa1297850d6e5b5039c3f751d36e2c76b8a150a5fa2936eee77f2ce7" exitCode=60
Oct 30 22:15:05.512048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.511935    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" event={"ID":"ec0991b6-ebfa-4f94-b400-df053996b279","Type":"ContainerDied","Data":"720ab40bfa1297850d6e5b5039c3f751d36e2c76b8a150a5fa2936eee77f2ce7"}
Oct 30 22:15:05.512048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.511989    4282 scope.go:122] "RemoveContainer" containerID="cfd594ffaa8ab216b7d88ac0b73e65b6f6f7b729dd081721eec0673d9c67f4ef"
Oct 30 22:15:05.512249 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.512232    4282 scope.go:122] "RemoveContainer" containerID="720ab40bfa1297850d6e5b5039c3f751d36e2c76b8a150a5fa2936eee77f2ce7"
Oct 30 22:15:05.512686 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:05.512660    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm_odh-ai-gateway-infra(ec0991b6-ebfa-4f94-b400-df053996b279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" podUID="ec0991b6-ebfa-4f94-b400-df053996b279"
Oct 30 22:15:05.514356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.514323    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" event={"ID":"02609366-36d2-492b-9aed-c0afc70b9b6e","Type":"ContainerStarted","Data":"aa830484cd8c12dbbec5d149e2edcacdc537663a296541a114c4ddb5ff5d394f"}
Oct 30 22:15:05.516307 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.516283    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" event={"ID":"65dc0398-a945-4341-893b-53cca23a17ea","Type":"ContainerStarted","Data":"71cdab5028094e9ce7710d825467f9171fab0f1240ef34171d53a8a8e1fe7ed5"}
Oct 30 22:15:05.518244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:05.518226    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" event={"ID":"f70b44c3-a9f0-4010-9ac0-5a4f42335537","Type":"ContainerStarted","Data":"6ad16f6d3057bb2b0078bf2fa097b50f14962e99e4dfd514734bb97ce0edcd5a"}
Oct 30 22:15:06.543931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.541378    4282 generic.go:356] "Generic (PLEG): container finished" podID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerID="37ea822ca2dd639864dbc3212b513b934dc820d8ca4d2353f378a6721da8557d" exitCode=60
Oct 30 22:15:06.543931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.541480    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" event={"ID":"a3b3fefe-386a-4d09-aad5-857c41bad582","Type":"ContainerDied","Data":"37ea822ca2dd639864dbc3212b513b934dc820d8ca4d2353f378a6721da8557d"}
Oct 30 22:15:06.543931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.541538    4282 scope.go:122] "RemoveContainer" containerID="57627f8c45a1c8f64e255acfc2935aac7e84d7bcf80b68e4cb634ea43e4689b8"
Oct 30 22:15:06.543931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.542075    4282 scope.go:122] "RemoveContainer" containerID="37ea822ca2dd639864dbc3212b513b934dc820d8ca4d2353f378a6721da8557d"
Oct 30 22:15:06.546005 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:06.545728    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c_odh-ai-gateway-infra(a3b3fefe-386a-4d09-aad5-857c41bad582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582"
Oct 30 22:15:06.547691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.547544    4282 scope.go:122] "RemoveContainer" containerID="720ab40bfa1297850d6e5b5039c3f751d36e2c76b8a150a5fa2936eee77f2ce7"
Oct 30 22:15:06.548019 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:06.547992    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm_odh-ai-gateway-infra(ec0991b6-ebfa-4f94-b400-df053996b279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" podUID="ec0991b6-ebfa-4f94-b400-df053996b279"
Oct 30 22:15:06.550237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.550208    4282 generic.go:356] "Generic (PLEG): container finished" podID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerID="aa830484cd8c12dbbec5d149e2edcacdc537663a296541a114c4ddb5ff5d394f" exitCode=60
Oct 30 22:15:06.550354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.550241    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" event={"ID":"02609366-36d2-492b-9aed-c0afc70b9b6e","Type":"ContainerDied","Data":"aa830484cd8c12dbbec5d149e2edcacdc537663a296541a114c4ddb5ff5d394f"}
Oct 30 22:15:06.550739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.550713    4282 scope.go:122] "RemoveContainer" containerID="aa830484cd8c12dbbec5d149e2edcacdc537663a296541a114c4ddb5ff5d394f"
Oct 30 22:15:06.551087 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:06.551064    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs_odh-ai-gateway-infra(02609366-36d2-492b-9aed-c0afc70b9b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e"
Oct 30 22:15:06.553251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.553225    4282 generic.go:356] "Generic (PLEG): container finished" podID="65dc0398-a945-4341-893b-53cca23a17ea" containerID="71cdab5028094e9ce7710d825467f9171fab0f1240ef34171d53a8a8e1fe7ed5" exitCode=60
Oct 30 22:15:06.553352 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.553310    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" event={"ID":"65dc0398-a945-4341-893b-53cca23a17ea","Type":"ContainerDied","Data":"71cdab5028094e9ce7710d825467f9171fab0f1240ef34171d53a8a8e1fe7ed5"}
Oct 30 22:15:06.553749 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.553726    4282 scope.go:122] "RemoveContainer" containerID="71cdab5028094e9ce7710d825467f9171fab0f1240ef34171d53a8a8e1fe7ed5"
Oct 30 22:15:06.554156 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:06.554129    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf_odh-ai-gateway-infra(65dc0398-a945-4341-893b-53cca23a17ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" podUID="65dc0398-a945-4341-893b-53cca23a17ea"
Oct 30 22:15:06.556059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.556029    4282 generic.go:356] "Generic (PLEG): container finished" podID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerID="6ad16f6d3057bb2b0078bf2fa097b50f14962e99e4dfd514734bb97ce0edcd5a" exitCode=60
Oct 30 22:15:06.556398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.556378    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" event={"ID":"f70b44c3-a9f0-4010-9ac0-5a4f42335537","Type":"ContainerDied","Data":"6ad16f6d3057bb2b0078bf2fa097b50f14962e99e4dfd514734bb97ce0edcd5a"}
Oct 30 22:15:06.556653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.556635    4282 scope.go:122] "RemoveContainer" containerID="6ad16f6d3057bb2b0078bf2fa097b50f14962e99e4dfd514734bb97ce0edcd5a"
Oct 30 22:15:06.557030 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:06.557007    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh_odh-ai-gateway-infra(f70b44c3-a9f0-4010-9ac0-5a4f42335537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537"
Oct 30 22:15:06.559364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.559340    4282 generic.go:356] "Generic (PLEG): container finished" podID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerID="1e315879552cf56560f2dff46ca108d2918f719578764463a16a9d8b0a4ff183" exitCode=60
Oct 30 22:15:06.559538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.559400    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" event={"ID":"94837e5c-a04e-47d7-8b11-85b4ec07cb87","Type":"ContainerDied","Data":"1e315879552cf56560f2dff46ca108d2918f719578764463a16a9d8b0a4ff183"}
Oct 30 22:15:06.559834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.559817    4282 scope.go:122] "RemoveContainer" containerID="1e315879552cf56560f2dff46ca108d2918f719578764463a16a9d8b0a4ff183"
Oct 30 22:15:06.560235 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:06.560211    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv_odh-ai-gateway-infra(94837e5c-a04e-47d7-8b11-85b4ec07cb87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87"
Oct 30 22:15:06.566171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.566139    4282 scope.go:122] "RemoveContainer" containerID="6b15f9b397eec19622a6fc3fb884298725e07762a565c08ed6e6518e2660dc18"
Oct 30 22:15:06.581538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.581522    4282 scope.go:122] "RemoveContainer" containerID="2c5a41b1ce9d441fcdd22780871da88860565ee2496fa768d74d5ab90fc80fc9"
Oct 30 22:15:06.596936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.596916    4282 scope.go:122] "RemoveContainer" containerID="f1ff99233eec06f93377513d4e698870f977d262ef36c604ff8d7ee8325d432e"
Oct 30 22:15:06.613388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:06.613353    4282 scope.go:122] "RemoveContainer" containerID="8e44499325a85fc815ce62e0d35e7d274ec94bef2274094d145083951f81da08"
Oct 30 22:15:07.570411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:07.570383    4282 scope.go:122] "RemoveContainer" containerID="1e315879552cf56560f2dff46ca108d2918f719578764463a16a9d8b0a4ff183"
Oct 30 22:15:07.570959 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:07.570839    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv_odh-ai-gateway-infra(94837e5c-a04e-47d7-8b11-85b4ec07cb87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87"
Oct 30 22:15:07.572405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:07.572373    4282 scope.go:122] "RemoveContainer" containerID="37ea822ca2dd639864dbc3212b513b934dc820d8ca4d2353f378a6721da8557d"
Oct 30 22:15:07.572854 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:07.572826    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c_odh-ai-gateway-infra(a3b3fefe-386a-4d09-aad5-857c41bad582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582"
Oct 30 22:15:07.574010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:07.573993    4282 scope.go:122] "RemoveContainer" containerID="aa830484cd8c12dbbec5d149e2edcacdc537663a296541a114c4ddb5ff5d394f"
Oct 30 22:15:07.574309 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:07.574284    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs_odh-ai-gateway-infra(02609366-36d2-492b-9aed-c0afc70b9b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e"
Oct 30 22:15:07.575638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:07.575621    4282 scope.go:122] "RemoveContainer" containerID="71cdab5028094e9ce7710d825467f9171fab0f1240ef34171d53a8a8e1fe7ed5"
Oct 30 22:15:07.575946 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:07.575928    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf_odh-ai-gateway-infra(65dc0398-a945-4341-893b-53cca23a17ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" podUID="65dc0398-a945-4341-893b-53cca23a17ea"
Oct 30 22:15:07.577631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:07.577614    4282 scope.go:122] "RemoveContainer" containerID="6ad16f6d3057bb2b0078bf2fa097b50f14962e99e4dfd514734bb97ce0edcd5a"
Oct 30 22:15:07.577866 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:07.577848    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh_odh-ai-gateway-infra(f70b44c3-a9f0-4010-9ac0-5a4f42335537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537"
Oct 30 22:15:08.263518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.263485    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"]
Oct 30 22:15:08.270209 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.270188    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.282499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.279403    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-28b4bb\""
Oct 30 22:15:08.282499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.279454    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"]
Oct 30 22:15:08.282499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.279559    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"routing-overlay\""
Oct 30 22:15:08.282499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.279638    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-813840a060cc\""
Oct 30 22:15:08.282499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.279705    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"kube-root-ca.crt\""
Oct 30 22:15:08.282499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.279748    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"payload-processing-external-model-e2e--eb82ae76-dockercfg-zdlx9\""
Oct 30 22:15:08.387555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.383490    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"agc-cred-6cef1b7cb072\""
Oct 30 22:15:08.395368 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.393083    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-28b4bb\"/\"openshift-service-ca.crt\""
Oct 30 22:15:08.480975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.480938    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.481304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.481279    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74xk4\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-kube-api-access-74xk4\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.481432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.481415    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.481571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.481555    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.582957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.582853    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74xk4\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-kube-api-access-74xk4\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.583314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.582973    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.583314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.583071    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.583314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.583097    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.584348 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.584315    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.584819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.584794    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.586784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.586759    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.599850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.599826    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74xk4\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-kube-api-access-74xk4\") pod \"payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") " pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:08.889710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:08.889637    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:09.243057 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:09.243021    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59cf758c_b4bb_4797_bcec_113d8262c26f.slice/crio-dd0e271c2005bfd141c2919c0e82204b02f2774f57ac6e2ea3443aa8ac6e2b2d WatchSource:0}: Error finding container dd0e271c2005bfd141c2919c0e82204b02f2774f57ac6e2ea3443aa8ac6e2b2d: Status 404 returned error can't find the container with id dd0e271c2005bfd141c2919c0e82204b02f2774f57ac6e2ea3443aa8ac6e2b2d
Oct 30 22:15:09.247065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:09.247038    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"]
Oct 30 22:15:09.606719 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:09.606626    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97" event={"ID":"59cf758c-b4bb-4797-bcec-113d8262c26f","Type":"ContainerStarted","Data":"de2069638a430664f643c8a5e1672d95f7a644afe24850c8c77cf7486984a3ff"}
Oct 30 22:15:09.606719 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:09.606683    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97" event={"ID":"59cf758c-b4bb-4797-bcec-113d8262c26f","Type":"ContainerStarted","Data":"dd0e271c2005bfd141c2919c0e82204b02f2774f57ac6e2ea3443aa8ac6e2b2d"}
Oct 30 22:15:09.607148 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:09.606723    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:09.627146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:09.627100    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97" podStartSLOduration=1.627086152 podStartE2EDuration="1.627086152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:09.62363235 +0000 UTC m=+2128.281728586" watchObservedRunningTime="2026-10-30 22:15:09.627086152 +0000 UTC m=+2128.285182374"
Oct 30 22:15:16.457790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:16.457760    4282 scope.go:122] "RemoveContainer" containerID="720ab40bfa1297850d6e5b5039c3f751d36e2c76b8a150a5fa2936eee77f2ce7"
Oct 30 22:15:17.457573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:17.457540    4282 scope.go:122] "RemoveContainer" containerID="71cdab5028094e9ce7710d825467f9171fab0f1240ef34171d53a8a8e1fe7ed5"
Oct 30 22:15:17.788235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:17.788151    4282 generic.go:356] "Generic (PLEG): container finished" podID="ec0991b6-ebfa-4f94-b400-df053996b279" containerID="e5b5e46b1484780c9cec7aac01aac01f5c1ac1410024a1217ba3d514bfed0e32" exitCode=60
Oct 30 22:15:17.789575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:17.788802    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" event={"ID":"ec0991b6-ebfa-4f94-b400-df053996b279","Type":"ContainerDied","Data":"e5b5e46b1484780c9cec7aac01aac01f5c1ac1410024a1217ba3d514bfed0e32"}
Oct 30 22:15:17.789575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:17.788953    4282 scope.go:122] "RemoveContainer" containerID="720ab40bfa1297850d6e5b5039c3f751d36e2c76b8a150a5fa2936eee77f2ce7"
Oct 30 22:15:17.790675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:17.789830    4282 scope.go:122] "RemoveContainer" containerID="e5b5e46b1484780c9cec7aac01aac01f5c1ac1410024a1217ba3d514bfed0e32"
Oct 30 22:15:17.790675 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:17.790230    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm_odh-ai-gateway-infra(ec0991b6-ebfa-4f94-b400-df053996b279)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" podUID="ec0991b6-ebfa-4f94-b400-df053996b279"
Oct 30 22:15:18.456843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:18.456772    4282 scope.go:122] "RemoveContainer" containerID="1e315879552cf56560f2dff46ca108d2918f719578764463a16a9d8b0a4ff183"
Oct 30 22:15:18.802719 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:18.802640    4282 generic.go:356] "Generic (PLEG): container finished" podID="65dc0398-a945-4341-893b-53cca23a17ea" containerID="b37ef51d959700ecd742c054d543b14da299d45c826b275da370f9e3413af415" exitCode=60
Oct 30 22:15:18.803164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:18.802714    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" event={"ID":"65dc0398-a945-4341-893b-53cca23a17ea","Type":"ContainerDied","Data":"b37ef51d959700ecd742c054d543b14da299d45c826b275da370f9e3413af415"}
Oct 30 22:15:18.803164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:18.802813    4282 scope.go:122] "RemoveContainer" containerID="71cdab5028094e9ce7710d825467f9171fab0f1240ef34171d53a8a8e1fe7ed5"
Oct 30 22:15:18.803308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:18.803292    4282 scope.go:122] "RemoveContainer" containerID="b37ef51d959700ecd742c054d543b14da299d45c826b275da370f9e3413af415"
Oct 30 22:15:18.803713 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:18.803684    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf_odh-ai-gateway-infra(65dc0398-a945-4341-893b-53cca23a17ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" podUID="65dc0398-a945-4341-893b-53cca23a17ea"
Oct 30 22:15:18.819742 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:18.819717    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"]
Oct 30 22:15:19.020793 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.020766    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:19.114052 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.113973    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0991b6-ebfa-4f94-b400-df053996b279-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0991b6-ebfa-4f94-b400-df053996b279-maas-api-service-ca\") pod \"ec0991b6-ebfa-4f94-b400-df053996b279\" (UID: \"ec0991b6-ebfa-4f94-b400-df053996b279\") "
Oct 30 22:15:19.114244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.114227    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0991b6-ebfa-4f94-b400-df053996b279-kube-api-access-qvxvj\" (UniqueName: \"kubernetes.io/projected/ec0991b6-ebfa-4f94-b400-df053996b279-kube-api-access-qvxvj\") pod \"ec0991b6-ebfa-4f94-b400-df053996b279\" (UID: \"ec0991b6-ebfa-4f94-b400-df053996b279\") "
Oct 30 22:15:19.114335 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.114311    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0991b6-ebfa-4f94-b400-df053996b279-maas-api-service-ca" pod "ec0991b6-ebfa-4f94-b400-df053996b279" (UID: "ec0991b6-ebfa-4f94-b400-df053996b279"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.114773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.114752    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec0991b6-ebfa-4f94-b400-df053996b279-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.116573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.116546    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0991b6-ebfa-4f94-b400-df053996b279-kube-api-access-qvxvj" pod "ec0991b6-ebfa-4f94-b400-df053996b279" (UID: "ec0991b6-ebfa-4f94-b400-df053996b279"). InnerVolumeSpecName "kube-api-access-qvxvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.216490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.216440    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvxvj\" (UniqueName: \"kubernetes.io/projected/ec0991b6-ebfa-4f94-b400-df053996b279-kube-api-access-qvxvj\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.456810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.456784    4282 scope.go:122] "RemoveContainer" containerID="6ad16f6d3057bb2b0078bf2fa097b50f14962e99e4dfd514734bb97ce0edcd5a"
Oct 30 22:15:19.457077 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.457027    4282 scope.go:122] "RemoveContainer" containerID="37ea822ca2dd639864dbc3212b513b934dc820d8ca4d2353f378a6721da8557d"
Oct 30 22:15:19.696356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.696321    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"]
Oct 30 22:15:19.697236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.697221    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:19.697294 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.697238    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:19.697294 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.697248    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:19.697294 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.697254    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:19.697405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.697395    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:19.697441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.697407    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:19.697441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.697420    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:19.702765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.702742    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.706014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.705961    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:15:19.712790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.712767    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"]
Oct 30 22:15:19.812685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.812658    4282 generic.go:356] "Generic (PLEG): container finished" podID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerID="d90cb09b47111548191fb30b40b5c3809015af881c6de365b7416b75a106ee1b" exitCode=60
Oct 30 22:15:19.813114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.812740    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" event={"ID":"94837e5c-a04e-47d7-8b11-85b4ec07cb87","Type":"ContainerDied","Data":"d90cb09b47111548191fb30b40b5c3809015af881c6de365b7416b75a106ee1b"}
Oct 30 22:15:19.813114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.812800    4282 scope.go:122] "RemoveContainer" containerID="1e315879552cf56560f2dff46ca108d2918f719578764463a16a9d8b0a4ff183"
Oct 30 22:15:19.813256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.813239    4282 scope.go:122] "RemoveContainer" containerID="d90cb09b47111548191fb30b40b5c3809015af881c6de365b7416b75a106ee1b"
Oct 30 22:15:19.813763 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:19.813735    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv_odh-ai-gateway-infra(94837e5c-a04e-47d7-8b11-85b4ec07cb87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87"
Oct 30 22:15:19.814417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.814386    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm" event={"ID":"ec0991b6-ebfa-4f94-b400-df053996b279","Type":"ContainerDied","Data":"ecbc157bae893af690b2184b1b553ad33d915aea2dcb0d513f8d7870e5b3173c"}
Oct 30 22:15:19.814529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.814401    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"
Oct 30 22:15:19.826875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.826858    4282 scope.go:122] "RemoveContainer" containerID="e5b5e46b1484780c9cec7aac01aac01f5c1ac1410024a1217ba3d514bfed0e32"
Oct 30 22:15:19.835222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.835200    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"]
Oct 30 22:15:19.836484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.836437    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69nfc\" (UniqueName: \"kubernetes.io/projected/95212ca9-d770-4bb4-9586-0509688b960d-kube-api-access-69nfc\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.836593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.836521    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.836706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.836689    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.854624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.854578    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"]
Oct 30 22:15:19.864330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.864304    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889975-4drsm"]
Oct 30 22:15:19.938293 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.938259    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.938730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.938704    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nfc\" (UniqueName: \"kubernetes.io/projected/95212ca9-d770-4bb4-9586-0509688b960d-kube-api-access-69nfc\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.938838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.938772    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.939519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.939494    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.943037 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.943013    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:19.951297 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:19.951275    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nfc\" (UniqueName: \"kubernetes.io/projected/95212ca9-d770-4bb4-9586-0509688b960d-kube-api-access-69nfc\") pod \"maas-api-revoke-sub-536e9795-4475f\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:20.026265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.026232    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:20.027089 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.027070    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:20.185383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.182082    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dc0398-a945-4341-893b-53cca23a17ea-kube-api-access-f4wxf\" (UniqueName: \"kubernetes.io/projected/65dc0398-a945-4341-893b-53cca23a17ea-kube-api-access-f4wxf\") pod \"65dc0398-a945-4341-893b-53cca23a17ea\" (UID: \"65dc0398-a945-4341-893b-53cca23a17ea\") "
Oct 30 22:15:20.185383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.182309    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dc0398-a945-4341-893b-53cca23a17ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc0398-a945-4341-893b-53cca23a17ea-maas-api-service-ca\") pod \"65dc0398-a945-4341-893b-53cca23a17ea\" (UID: \"65dc0398-a945-4341-893b-53cca23a17ea\") "
Oct 30 22:15:20.185383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.182807    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dc0398-a945-4341-893b-53cca23a17ea-maas-api-service-ca" pod "65dc0398-a945-4341-893b-53cca23a17ea" (UID: "65dc0398-a945-4341-893b-53cca23a17ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.226494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.214624    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65dc0398-a945-4341-893b-53cca23a17ea-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.247318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.246973    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dc0398-a945-4341-893b-53cca23a17ea-kube-api-access-f4wxf" pod "65dc0398-a945-4341-893b-53cca23a17ea" (UID: "65dc0398-a945-4341-893b-53cca23a17ea"). InnerVolumeSpecName "kube-api-access-f4wxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.316029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.315994    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4wxf\" (UniqueName: \"kubernetes.io/projected/65dc0398-a945-4341-893b-53cca23a17ea-kube-api-access-f4wxf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.327490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.327450    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"]
Oct 30 22:15:20.328310 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:20.328279    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95212ca9_d770_4bb4_9586_0509688b960d.slice/crio-62960b4e99b8921d5b69ae0bf88b6bc86b83d4e80dd1b7bdc45885a27bbc9b3f WatchSource:0}: Error finding container 62960b4e99b8921d5b69ae0bf88b6bc86b83d4e80dd1b7bdc45885a27bbc9b3f: Status 404 returned error can't find the container with id 62960b4e99b8921d5b69ae0bf88b6bc86b83d4e80dd1b7bdc45885a27bbc9b3f
Oct 30 22:15:20.457417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.457390    4282 scope.go:122] "RemoveContainer" containerID="aa830484cd8c12dbbec5d149e2edcacdc537663a296541a114c4ddb5ff5d394f"
Oct 30 22:15:20.618780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.618757    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:15:20.910355 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.910318    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"]
Oct 30 22:15:20.934152 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.934124    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"
Oct 30 22:15:20.934542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.934123    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf" event={"ID":"65dc0398-a945-4341-893b-53cca23a17ea","Type":"ContainerDied","Data":"d5536268ad4e4cf333c85bac14c72daaa961ac61c5ed75ab814559c2b4b3efe3"}
Oct 30 22:15:20.934621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.934583    4282 scope.go:122] "RemoveContainer" containerID="b37ef51d959700ecd742c054d543b14da299d45c826b275da370f9e3413af415"
Oct 30 22:15:20.937220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.937186    4282 generic.go:356] "Generic (PLEG): container finished" podID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerID="4fd5555acd88f679c98bc1e08c1c47a019effb1e59574c0120143af68f27601d" exitCode=60
Oct 30 22:15:20.937220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.937225    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" event={"ID":"f70b44c3-a9f0-4010-9ac0-5a4f42335537","Type":"ContainerDied","Data":"4fd5555acd88f679c98bc1e08c1c47a019effb1e59574c0120143af68f27601d"}
Oct 30 22:15:20.938865 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.937782    4282 scope.go:122] "RemoveContainer" containerID="4fd5555acd88f679c98bc1e08c1c47a019effb1e59574c0120143af68f27601d"
Oct 30 22:15:20.938865 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:20.938135    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh_odh-ai-gateway-infra(f70b44c3-a9f0-4010-9ac0-5a4f42335537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537"
Oct 30 22:15:20.941312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.941283    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f" event={"ID":"95212ca9-d770-4bb4-9586-0509688b960d","Type":"ContainerStarted","Data":"ff40b2e3a2690a10ba9fdb606b50cc5682cfd5e6916633a722bdb5045d8a8dc5"}
Oct 30 22:15:20.941438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.941337    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f" event={"ID":"95212ca9-d770-4bb4-9586-0509688b960d","Type":"ContainerStarted","Data":"62960b4e99b8921d5b69ae0bf88b6bc86b83d4e80dd1b7bdc45885a27bbc9b3f"}
Oct 30 22:15:20.943565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.943542    4282 generic.go:356] "Generic (PLEG): container finished" podID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerID="15ff6894e9c12e4cc056f9a6f17cd4cbbd1c4bb0bb06bea70820556bec6d48cc" exitCode=60
Oct 30 22:15:20.943682 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.943606    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" event={"ID":"a3b3fefe-386a-4d09-aad5-857c41bad582","Type":"ContainerDied","Data":"15ff6894e9c12e4cc056f9a6f17cd4cbbd1c4bb0bb06bea70820556bec6d48cc"}
Oct 30 22:15:20.945032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.944016    4282 scope.go:122] "RemoveContainer" containerID="15ff6894e9c12e4cc056f9a6f17cd4cbbd1c4bb0bb06bea70820556bec6d48cc"
Oct 30 22:15:20.945032 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:20.944272    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c_odh-ai-gateway-infra(a3b3fefe-386a-4d09-aad5-857c41bad582)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582"
Oct 30 22:15:20.950087 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.950066    4282 scope.go:122] "RemoveContainer" containerID="6ad16f6d3057bb2b0078bf2fa097b50f14962e99e4dfd514734bb97ce0edcd5a"
Oct 30 22:15:20.967097 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.967047    4282 scope.go:122] "RemoveContainer" containerID="37ea822ca2dd639864dbc3212b513b934dc820d8ca4d2353f378a6721da8557d"
Oct 30 22:15:20.986699 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.986650    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"]
Oct 30 22:15:20.992815 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:20.992788    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889975-dstgf"]
Oct 30 22:15:21.053916 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.053855    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f" podStartSLOduration=2.053836056 podStartE2EDuration="2.053836056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:21.029788995 +0000 UTC m=+2139.687885221" watchObservedRunningTime="2026-10-30 22:15:21.053836056 +0000 UTC m=+2139.711932279"
Oct 30 22:15:21.128767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.128741    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:21.213341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.213310    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94837e5c-a04e-47d7-8b11-85b4ec07cb87-kube-api-access-5zmw2\" (UniqueName: \"kubernetes.io/projected/94837e5c-a04e-47d7-8b11-85b4ec07cb87-kube-api-access-5zmw2\") pod \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\" (UID: \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\") "
Oct 30 22:15:21.213714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.213694    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94837e5c-a04e-47d7-8b11-85b4ec07cb87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94837e5c-a04e-47d7-8b11-85b4ec07cb87-maas-api-service-ca\") pod \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\" (UID: \"94837e5c-a04e-47d7-8b11-85b4ec07cb87\") "
Oct 30 22:15:21.214040 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.214019    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94837e5c-a04e-47d7-8b11-85b4ec07cb87-maas-api-service-ca" pod "94837e5c-a04e-47d7-8b11-85b4ec07cb87" (UID: "94837e5c-a04e-47d7-8b11-85b4ec07cb87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.214300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.214281    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94837e5c-a04e-47d7-8b11-85b4ec07cb87-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.215675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.215647    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94837e5c-a04e-47d7-8b11-85b4ec07cb87-kube-api-access-5zmw2" pod "94837e5c-a04e-47d7-8b11-85b4ec07cb87" (UID: "94837e5c-a04e-47d7-8b11-85b4ec07cb87"). InnerVolumeSpecName "kube-api-access-5zmw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.316167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.316135    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmw2\" (UniqueName: \"kubernetes.io/projected/94837e5c-a04e-47d7-8b11-85b4ec07cb87-kube-api-access-5zmw2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.470553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.470440    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dc0398-a945-4341-893b-53cca23a17ea" path="/var/lib/kubelet/pods/65dc0398-a945-4341-893b-53cca23a17ea/volumes"
Oct 30 22:15:21.471170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.471150    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" path="/var/lib/kubelet/pods/ec0991b6-ebfa-4f94-b400-df053996b279/volumes"
Oct 30 22:15:21.955345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.955310    4282 generic.go:356] "Generic (PLEG): container finished" podID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerID="d50e8a9003ec569267cd0c7cf85082385f3d2dc06dfdfa4bed5ff2eace81ed5b" exitCode=60
Oct 30 22:15:21.955843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.955386    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" event={"ID":"02609366-36d2-492b-9aed-c0afc70b9b6e","Type":"ContainerDied","Data":"d50e8a9003ec569267cd0c7cf85082385f3d2dc06dfdfa4bed5ff2eace81ed5b"}
Oct 30 22:15:21.955843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.955461    4282 scope.go:122] "RemoveContainer" containerID="aa830484cd8c12dbbec5d149e2edcacdc537663a296541a114c4ddb5ff5d394f"
Oct 30 22:15:21.955990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.955974    4282 scope.go:122] "RemoveContainer" containerID="d50e8a9003ec569267cd0c7cf85082385f3d2dc06dfdfa4bed5ff2eace81ed5b"
Oct 30 22:15:21.956780 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:21.956350    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs_odh-ai-gateway-infra(02609366-36d2-492b-9aed-c0afc70b9b6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e"
Oct 30 22:15:21.959968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.959944    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv" event={"ID":"94837e5c-a04e-47d7-8b11-85b4ec07cb87","Type":"ContainerDied","Data":"720e425b832dcb91eba8a408db32c4fd87f9b4039a8f692cc2476e29e14af6e8"}
Oct 30 22:15:21.959968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.959957    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"
Oct 30 22:15:21.961876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.961852    4282 generic.go:356] "Generic (PLEG): container finished" podID="95212ca9-d770-4bb4-9586-0509688b960d" containerID="ff40b2e3a2690a10ba9fdb606b50cc5682cfd5e6916633a722bdb5045d8a8dc5" exitCode=0
Oct 30 22:15:21.962003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.961926    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f" event={"ID":"95212ca9-d770-4bb4-9586-0509688b960d","Type":"ContainerDied","Data":"ff40b2e3a2690a10ba9fdb606b50cc5682cfd5e6916633a722bdb5045d8a8dc5"}
Oct 30 22:15:21.970600 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.970586    4282 scope.go:122] "RemoveContainer" containerID="d90cb09b47111548191fb30b40b5c3809015af881c6de365b7416b75a106ee1b"
Oct 30 22:15:21.971353 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.971334    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"]
Oct 30 22:15:21.987391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.987368    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"]
Oct 30 22:15:21.998664 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:21.998641    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889975-fjclv"]
Oct 30 22:15:22.065111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.065078    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"]
Oct 30 22:15:22.189196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.189170    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:22.244722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.243328    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70b44c3-a9f0-4010-9ac0-5a4f42335537-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b44c3-a9f0-4010-9ac0-5a4f42335537-maas-api-service-ca\") pod \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\" (UID: \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\") "
Oct 30 22:15:22.244722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.243452    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70b44c3-a9f0-4010-9ac0-5a4f42335537-kube-api-access-qzcxr\" (UniqueName: \"kubernetes.io/projected/f70b44c3-a9f0-4010-9ac0-5a4f42335537-kube-api-access-qzcxr\") pod \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\" (UID: \"f70b44c3-a9f0-4010-9ac0-5a4f42335537\") "
Oct 30 22:15:22.257948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.254253    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70b44c3-a9f0-4010-9ac0-5a4f42335537-maas-api-service-ca" pod "f70b44c3-a9f0-4010-9ac0-5a4f42335537" (UID: "f70b44c3-a9f0-4010-9ac0-5a4f42335537"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:22.310217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.310128    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70b44c3-a9f0-4010-9ac0-5a4f42335537-kube-api-access-qzcxr" pod "f70b44c3-a9f0-4010-9ac0-5a4f42335537" (UID: "f70b44c3-a9f0-4010-9ac0-5a4f42335537"). InnerVolumeSpecName "kube-api-access-qzcxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:22.344996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.344969    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:22.350908 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.350875    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:15:22.426676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.426610    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b3fefe-386a-4d09-aad5-857c41bad582-kube-api-access-8gwz5\" (UniqueName: \"kubernetes.io/projected/a3b3fefe-386a-4d09-aad5-857c41bad582-kube-api-access-8gwz5\") pod \"a3b3fefe-386a-4d09-aad5-857c41bad582\" (UID: \"a3b3fefe-386a-4d09-aad5-857c41bad582\") "
Oct 30 22:15:22.426895 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.426784    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b3fefe-386a-4d09-aad5-857c41bad582-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b3fefe-386a-4d09-aad5-857c41bad582-maas-api-service-ca\") pod \"a3b3fefe-386a-4d09-aad5-857c41bad582\" (UID: \"a3b3fefe-386a-4d09-aad5-857c41bad582\") "
Oct 30 22:15:22.427543 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.427513    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b3fefe-386a-4d09-aad5-857c41bad582-maas-api-service-ca" pod "a3b3fefe-386a-4d09-aad5-857c41bad582" (UID: "a3b3fefe-386a-4d09-aad5-857c41bad582"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:22.427690 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.427675    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70b44c3-a9f0-4010-9ac0-5a4f42335537-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.427767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.427696    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3b3fefe-386a-4d09-aad5-857c41bad582-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.427767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.427710    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcxr\" (UniqueName: \"kubernetes.io/projected/f70b44c3-a9f0-4010-9ac0-5a4f42335537-kube-api-access-qzcxr\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.430273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.430243    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b3fefe-386a-4d09-aad5-857c41bad582-kube-api-access-8gwz5" pod "a3b3fefe-386a-4d09-aad5-857c41bad582" (UID: "a3b3fefe-386a-4d09-aad5-857c41bad582"). InnerVolumeSpecName "kube-api-access-8gwz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:22.529814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.529783    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gwz5\" (UniqueName: \"kubernetes.io/projected/a3b3fefe-386a-4d09-aad5-857c41bad582-kube-api-access-8gwz5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.977365 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.977308    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh" event={"ID":"f70b44c3-a9f0-4010-9ac0-5a4f42335537","Type":"ContainerDied","Data":"ef459bda33e8fc34cf8f387369d98460ee0dd39cb4c36284e64c08dfe056517d"}
Oct 30 22:15:22.977903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.977382    4282 scope.go:122] "RemoveContainer" containerID="4fd5555acd88f679c98bc1e08c1c47a019effb1e59574c0120143af68f27601d"
Oct 30 22:15:22.977903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.977506    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"
Oct 30 22:15:22.985123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.985101    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"
Oct 30 22:15:22.985371 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.985129    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c" event={"ID":"a3b3fefe-386a-4d09-aad5-857c41bad582","Type":"ContainerDied","Data":"d2933a7d8eacc28c22bdcd5b4091a09b66ad99d9e18159b5e321b3da30268a11"}
Oct 30 22:15:22.992327 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.992303    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"]
Oct 30 22:15:22.998882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:22.998861    4282 scope.go:122] "RemoveContainer" containerID="15ff6894e9c12e4cc056f9a6f17cd4cbbd1c4bb0bb06bea70820556bec6d48cc"
Oct 30 22:15:23.017202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.017170    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"]
Oct 30 22:15:23.032366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.032297    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889975-ff7bh"]
Oct 30 22:15:23.080706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.080668    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"]
Oct 30 22:15:23.088729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.088329    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889975-stm5c"]
Oct 30 22:15:23.637266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.637230    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" path="/var/lib/kubelet/pods/94837e5c-a04e-47d7-8b11-85b4ec07cb87/volumes"
Oct 30 22:15:23.638337 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.638312    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" path="/var/lib/kubelet/pods/a3b3fefe-386a-4d09-aad5-857c41bad582/volumes"
Oct 30 22:15:23.639012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.638988    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" path="/var/lib/kubelet/pods/f70b44c3-a9f0-4010-9ac0-5a4f42335537/volumes"
Oct 30 22:15:23.702773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.702748    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:23.704873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.704845    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:23.860421 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.860345    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02609366-36d2-492b-9aed-c0afc70b9b6e-kube-api-access-5zh5n\" (UniqueName: \"kubernetes.io/projected/02609366-36d2-492b-9aed-c0afc70b9b6e-kube-api-access-5zh5n\") pod \"02609366-36d2-492b-9aed-c0afc70b9b6e\" (UID: \"02609366-36d2-492b-9aed-c0afc70b9b6e\") "
Oct 30 22:15:23.860616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.860524    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-xks-ca\") pod \"95212ca9-d770-4bb4-9586-0509688b960d\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") "
Oct 30 22:15:23.860616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.860572    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02609366-36d2-492b-9aed-c0afc70b9b6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02609366-36d2-492b-9aed-c0afc70b9b6e-maas-api-service-ca\") pod \"02609366-36d2-492b-9aed-c0afc70b9b6e\" (UID: \"02609366-36d2-492b-9aed-c0afc70b9b6e\") "
Oct 30 22:15:23.860729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.860648    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95212ca9-d770-4bb4-9586-0509688b960d-kube-api-access-69nfc\" (UniqueName: \"kubernetes.io/projected/95212ca9-d770-4bb4-9586-0509688b960d-kube-api-access-69nfc\") pod \"95212ca9-d770-4bb4-9586-0509688b960d\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") "
Oct 30 22:15:23.860804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.860748    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-openshift-ca\") pod \"95212ca9-d770-4bb4-9586-0509688b960d\" (UID: \"95212ca9-d770-4bb4-9586-0509688b960d\") "
Oct 30 22:15:23.861054 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.861026    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02609366-36d2-492b-9aed-c0afc70b9b6e-maas-api-service-ca" pod "02609366-36d2-492b-9aed-c0afc70b9b6e" (UID: "02609366-36d2-492b-9aed-c0afc70b9b6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:23.861774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.861739    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02609366-36d2-492b-9aed-c0afc70b9b6e-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.862069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.861772    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-openshift-ca" pod "95212ca9-d770-4bb4-9586-0509688b960d" (UID: "95212ca9-d770-4bb4-9586-0509688b960d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:23.863293 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.863269    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02609366-36d2-492b-9aed-c0afc70b9b6e-kube-api-access-5zh5n" pod "02609366-36d2-492b-9aed-c0afc70b9b6e" (UID: "02609366-36d2-492b-9aed-c0afc70b9b6e"). InnerVolumeSpecName "kube-api-access-5zh5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:23.863406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.863268    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-xks-ca" pod "95212ca9-d770-4bb4-9586-0509688b960d" (UID: "95212ca9-d770-4bb4-9586-0509688b960d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:23.863509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.863491    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95212ca9-d770-4bb4-9586-0509688b960d-kube-api-access-69nfc" pod "95212ca9-d770-4bb4-9586-0509688b960d" (UID: "95212ca9-d770-4bb4-9586-0509688b960d"). InnerVolumeSpecName "kube-api-access-69nfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:23.984488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.983499    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-openshift-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.984488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.983542    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zh5n\" (UniqueName: \"kubernetes.io/projected/02609366-36d2-492b-9aed-c0afc70b9b6e-kube-api-access-5zh5n\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.984488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.983575    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95212ca9-d770-4bb4-9586-0509688b960d-maas-api-xks-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:23.984488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:23.983592    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69nfc\" (UniqueName: \"kubernetes.io/projected/95212ca9-d770-4bb4-9586-0509688b960d-kube-api-access-69nfc\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:24.042636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:24.042599    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f" event={"ID":"95212ca9-d770-4bb4-9586-0509688b960d","Type":"ContainerDied","Data":"62960b4e99b8921d5b69ae0bf88b6bc86b83d4e80dd1b7bdc45885a27bbc9b3f"}
Oct 30 22:15:24.043038 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:24.043020    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62960b4e99b8921d5b69ae0bf88b6bc86b83d4e80dd1b7bdc45885a27bbc9b3f"
Oct 30 22:15:24.043210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:24.042929    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"
Oct 30 22:15:24.049699 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:24.049671    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs" event={"ID":"02609366-36d2-492b-9aed-c0afc70b9b6e","Type":"ContainerDied","Data":"4e1b8142ad9c843f8941a45420c37f3ba064303065e9dce64f48221750298cd7"}
Oct 30 22:15:24.050736 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:24.050718    4282 scope.go:122] "RemoveContainer" containerID="d50e8a9003ec569267cd0c7cf85082385f3d2dc06dfdfa4bed5ff2eace81ed5b"
Oct 30 22:15:24.050956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:24.050243    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"
Oct 30 22:15:24.248980 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:24.248915    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"]
Oct 30 22:15:24.257404 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:24.257376    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b2e72-29889975-zr5fs"]
Oct 30 22:15:25.465834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:25.465802    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" path="/var/lib/kubelet/pods/02609366-36d2-492b-9aed-c0afc70b9b6e/volumes"
Oct 30 22:15:31.124785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:31.124749    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"]
Oct 30 22:15:34.650780 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:34.650743    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:15:36.125494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.125406    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh" podUID="23ad18a4-c94a-455b-b097-ad268e10f623" containerName="payload-processing" containerID="cri-o://80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176" gracePeriod=25
Oct 30 22:15:36.479509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.479452    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:15:36.601338 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.601297    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-kube-api-access-gdzxw\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-kube-api-access-gdzxw\") pod \"23ad18a4-c94a-455b-b097-ad268e10f623\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") "
Oct 30 22:15:36.601569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.601355    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-routing-overlay-volume\") pod \"23ad18a4-c94a-455b-b097-ad268e10f623\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") "
Oct 30 22:15:36.601569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.601422    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-provider-credentials\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-provider-credentials\") pod \"23ad18a4-c94a-455b-b097-ad268e10f623\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") "
Oct 30 22:15:36.601841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.601815    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-plugins-config-volume\") pod \"23ad18a4-c94a-455b-b097-ad268e10f623\" (UID: \"23ad18a4-c94a-455b-b097-ad268e10f623\") "
Oct 30 22:15:36.601841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.601825    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-routing-overlay-volume" pod "23ad18a4-c94a-455b-b097-ad268e10f623" (UID: "23ad18a4-c94a-455b-b097-ad268e10f623"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:36.602272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.602249    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-plugins-config-volume" pod "23ad18a4-c94a-455b-b097-ad268e10f623" (UID: "23ad18a4-c94a-455b-b097-ad268e10f623"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:36.602774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.602750    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.602873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.602778    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/23ad18a4-c94a-455b-b097-ad268e10f623-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.603776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.603751    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-kube-api-access-gdzxw" pod "23ad18a4-c94a-455b-b097-ad268e10f623" (UID: "23ad18a4-c94a-455b-b097-ad268e10f623"). InnerVolumeSpecName "kube-api-access-gdzxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.603869 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.603810    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-provider-credentials" pod "23ad18a4-c94a-455b-b097-ad268e10f623" (UID: "23ad18a4-c94a-455b-b097-ad268e10f623"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.704827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.704739    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.704827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:36.704768    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdzxw\" (UniqueName: \"kubernetes.io/projected/23ad18a4-c94a-455b-b097-ad268e10f623-kube-api-access-gdzxw\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:37.231811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.231775    4282 generic.go:356] "Generic (PLEG): container finished" podID="23ad18a4-c94a-455b-b097-ad268e10f623" containerID="80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176" exitCode=0
Oct 30 22:15:37.232321 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.231827    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"
Oct 30 22:15:37.232321 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.231860    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh" event={"ID":"23ad18a4-c94a-455b-b097-ad268e10f623","Type":"ContainerDied","Data":"80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176"}
Oct 30 22:15:37.232321 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.231912    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh" event={"ID":"23ad18a4-c94a-455b-b097-ad268e10f623","Type":"ContainerDied","Data":"556cea034f9c1495d5812e276af32cfe18a0319ab15369d797f5849eac827f97"}
Oct 30 22:15:37.232321 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.231936    4282 scope.go:122] "RemoveContainer" containerID="80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176"
Oct 30 22:15:37.244283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.244267    4282 scope.go:122] "RemoveContainer" containerID="80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176"
Oct 30 22:15:37.244597 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:15:37.244580    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176\": container with ID starting with 80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176 not found: ID does not exist" containerID="80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176"
Oct 30 22:15:37.244655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.244607    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176"} err="failed to get container status \"80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176\": rpc error: code = NotFound desc = could not find container \"80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176\": container with ID starting with 80f81128e4a1bc492cad7851c9a356974163133133819f26738f0310c6926176 not found: ID does not exist"
Oct 30 22:15:37.258834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.258813    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"]
Oct 30 22:15:37.263296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.263276    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-26a128/payload-processing-external-model-e2e-extmod-b-26a128-5955hk4bh"]
Oct 30 22:15:37.466924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:37.466883    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ad18a4-c94a-455b-b097-ad268e10f623" path="/var/lib/kubelet/pods/23ad18a4-c94a-455b-b097-ad268e10f623/volumes"
Oct 30 22:15:57.590702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.590665    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"]
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591891    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591908    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591928    4282 cpu_manager.go:414] "containerMap: removing container" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591935    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591948    4282 cpu_manager.go:414] "containerMap: removing container" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591956    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591968    4282 cpu_manager.go:414] "containerMap: removing container" podUID="23ad18a4-c94a-455b-b097-ad268e10f623" containerName="payload-processing"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591975    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ad18a4-c94a-455b-b097-ad268e10f623" containerName="payload-processing"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591986    4282 cpu_manager.go:414] "containerMap: removing container" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.591992    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592004    4282 cpu_manager.go:414] "containerMap: removing container" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592011    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592029    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592036    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592045    4282 cpu_manager.go:414] "containerMap: removing container" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592054    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592062    4282 cpu_manager.go:414] "containerMap: removing container" podUID="95212ca9-d770-4bb4-9586-0509688b960d" containerName="revoke-keys"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592069    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95212ca9-d770-4bb4-9586-0509688b960d" containerName="revoke-keys"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592086    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592093    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592102    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592108    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592120    4282 cpu_manager.go:414] "containerMap: removing container" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592126    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592137    4282 cpu_manager.go:414] "containerMap: removing container" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592143    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592156    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592163    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592182    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592188    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592201    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592207    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0991b6-ebfa-4f94-b400-df053996b279" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592236    4282 cpu_manager.go:414] "containerMap: removing container" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592243    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592451    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592480    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592489    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592498    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ad18a4-c94a-455b-b097-ad268e10f623" containerName="payload-processing"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592509    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592516    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592524    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592531    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592538    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="95212ca9-d770-4bb4-9586-0509688b960d" containerName="revoke-keys"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592549    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dc0398-a945-4341-893b-53cca23a17ea" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592560    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592568    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:15:57.593073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.592580    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:15:57.597194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.597172    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.603991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.603967    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"]
Oct 30 22:15:57.605439 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.605136    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"agc-cred-cb4046ae8a86\""
Oct 30 22:15:57.605618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.605273    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6f8222\"/\"agc-cred-940fe11d813a\""
Oct 30 22:15:57.734771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.734731    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.734976 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.734962    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7hc\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-kube-api-access-8m7hc\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.735213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.735199    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.735344 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.735334    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.836849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.836755    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.836849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.836822    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7hc\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-kube-api-access-8m7hc\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.837087 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.836965    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.837087 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.837014    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.837688 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.837661    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.837804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.837769    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.839584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.839558    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.848313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.848263    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7hc\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-kube-api-access-8m7hc\") pod \"payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") " pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:57.966828 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:57.966795    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:58.260982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.260957    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"]
Oct 30 22:15:58.261548 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:15:58.261519    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3b25d4_5b07_4c09_9670_a772432bd313.slice/crio-a78cd7a8e4e661e8066eedd4c4d5771355227f0cafe5d186749ce0a094e78b76 WatchSource:0}: Error finding container a78cd7a8e4e661e8066eedd4c4d5771355227f0cafe5d186749ce0a094e78b76: Status 404 returned error can't find the container with id a78cd7a8e4e661e8066eedd4c4d5771355227f0cafe5d186749ce0a094e78b76
Oct 30 22:15:58.741843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.741810    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr" event={"ID":"ab3b25d4-5b07-4c09-9670-a772432bd313","Type":"ContainerStarted","Data":"9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73"}
Oct 30 22:15:58.742304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.741907    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:15:58.742304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.741934    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr" event={"ID":"ab3b25d4-5b07-4c09-9670-a772432bd313","Type":"ContainerStarted","Data":"a78cd7a8e4e661e8066eedd4c4d5771355227f0cafe5d186749ce0a094e78b76"}
Oct 30 22:15:58.743780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.743754    4282 generic.go:356] "Generic (PLEG): container finished" podID="579717b1-feef-4e2a-9da8-62721859adeb" containerID="8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d" exitCode=0
Oct 30 22:15:58.743883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.743825    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" event={"ID":"579717b1-feef-4e2a-9da8-62721859adeb","Type":"ContainerDied","Data":"8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d"}
Oct 30 22:15:58.744271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.744257    4282 scope.go:122] "RemoveContainer" containerID="8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d"
Oct 30 22:15:58.746568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.746537    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:15:58.746655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.746642    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:15:58.746718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.746702    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:15:58.762215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:58.762168    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr" podStartSLOduration=1.7621523639999999 podStartE2EDuration="1.762152364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:58.760289243 +0000 UTC m=+2177.418385469" watchObservedRunningTime="2026-10-30 22:15:58.762152364 +0000 UTC m=+2177.420248587"
Oct 30 22:15:59.753064 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:15:59.753026    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" event={"ID":"579717b1-feef-4e2a-9da8-62721859adeb","Type":"ContainerStarted","Data":"b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d"}
Oct 30 22:16:00.634492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:00.632744    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:16:00.634492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:00.632915    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:16:00.634492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:00.633004    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 22:16:01.623692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:01.623659    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:16:01.624130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:01.623706    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 22:16:02.623773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:02.623739    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:16:02.624395 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:02.624351    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 22:16:03.624336 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:03.624295    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:16:03.624866 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:03.624354    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 22:16:04.624111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:04.624081    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:16:04.624303 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:04.624141    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 22:16:04.651341 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:16:04.651308    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:16:05.039534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.025695    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"]
Oct 30 22:16:05.041237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.041185    4282 cpu_manager.go:414] "containerMap: removing container" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:16:05.041237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.041221    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:16:05.041643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.041434    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b3fefe-386a-4d09-aad5-857c41bad582" containerName="cleanup"
Oct 30 22:16:05.041643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.041454    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="94837e5c-a04e-47d7-8b11-85b4ec07cb87" containerName="cleanup"
Oct 30 22:16:05.041643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.041491    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70b44c3-a9f0-4010-9ac0-5a4f42335537" containerName="cleanup"
Oct 30 22:16:05.041643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.041502    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="02609366-36d2-492b-9aed-c0afc70b9b6e" containerName="cleanup"
Oct 30 22:16:05.066909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.066842    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.093656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.093611    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"]
Oct 30 22:16:05.172883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.172841    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.173222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.173202    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.173570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.173547    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.173874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.173854    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.174053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.174036    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.174325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.174304    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.174519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.174502    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2qx\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-kube-api-access-vf2qx\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.174720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.174702    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.174881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.174865    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.277579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.277530    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.277853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.277603    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.277853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.277635    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.277853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.277812    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.277853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.277848    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278002    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278037    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2qx\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-kube-api-access-vf2qx\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278101    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278126    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278146    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278444    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278457    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278555    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.278762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.278737    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.280960 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.280933    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.281251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.281232    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.293724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.293671    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.293841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.293825    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2qx\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-kube-api-access-vf2qx\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.419146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.419118    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:05.624074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.623974    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:16:05.624074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.624039    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 22:16:05.829342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.829310    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"]
Oct 30 22:16:05.830242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.830204    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:16:05.830377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.830298    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:16:05.830377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.830345    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:16:05.841729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:05.841703    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" event={"ID":"ba075b75-0bd2-45db-b1d7-a2b61edd6508","Type":"ContainerStarted","Data":"798a295c4afb28bef4211d8527822e454262fad21fc9ca32ff47ffa446f1f8d7"}
Oct 30 22:16:06.670827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:06.670794    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:16:06.680815 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:06.677017    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:16:06.689936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:06.689902    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:16:06.852298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:06.852246    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" event={"ID":"ba075b75-0bd2-45db-b1d7-a2b61edd6508","Type":"ContainerStarted","Data":"f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b"}
Oct 30 22:16:06.899080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:06.899021    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podStartSLOduration=2.899002513 podStartE2EDuration="2.899002513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:06.877817326 +0000 UTC m=+2185.535913548" watchObservedRunningTime="2026-10-30 22:16:06.899002513 +0000 UTC m=+2185.557098735"
Oct 30 22:16:07.419397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:07.419366    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:07.421403 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:07.421377    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:16:07.421589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:07.421438    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 22:16:08.579490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:08.577939    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:16:08.579490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:08.578008    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 22:16:09.114448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.114414    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"]
Oct 30 22:16:09.120422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.120403    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.125244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.125219    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bacea0\""
Oct 30 22:16:09.125397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.125270    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"kube-root-ca.crt\""
Oct 30 22:16:09.125397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.125345    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"agc-cred-49b48f4d8815\""
Oct 30 22:16:09.125397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.125362    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"openshift-service-ca.crt\""
Oct 30 22:16:09.125397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.125383    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"routing-overlay\""
Oct 30 22:16:09.125714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.125538    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bacea0\"/\"payload-processing-external-model-e2e--6ed667c1-dockercfg-sm96s\""
Oct 30 22:16:09.132763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.132741    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"]
Oct 30 22:16:09.394555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.370643    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpxt\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-kube-api-access-5mpxt\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.419163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.419126    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.419916 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.419895    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.532497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.505292    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.532990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.419838    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:16:09.544886 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.544842    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 22:16:09.634629 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.634595    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpxt\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-kube-api-access-5mpxt\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.635459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.635429    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.635667 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.635646    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.635929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.635909    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.636242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.636218    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.636580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.636559    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.638242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.638223    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.648172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.648040    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpxt\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-kube-api-access-5mpxt\") pod \"payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") " pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.737994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.737963    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:09.784304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.783325    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:16:09.850910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.850723    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"]
Oct 30 22:16:09.920539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:09.920509    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:16:10.110154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:10.110116    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"]
Oct 30 22:16:10.111153 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:16:10.111126    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb574051_68be_4a1a_8c3a_daae2a838fa2.slice/crio-66fe9276f320ef749d9c54598bac8d659b316338d0e772f2ce913d187b50ae99 WatchSource:0}: Error finding container 66fe9276f320ef749d9c54598bac8d659b316338d0e772f2ce913d187b50ae99: Status 404 returned error can't find the container with id 66fe9276f320ef749d9c54598bac8d659b316338d0e772f2ce913d187b50ae99
Oct 30 22:16:10.420458 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:10.420417    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:16:10.420673 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:10.420498    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 22:16:11.083501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:11.082180    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl" event={"ID":"fb574051-68be-4a1a-8c3a-daae2a838fa2","Type":"ContainerStarted","Data":"5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3"}
Oct 30 22:16:11.083501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:11.082547    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl" event={"ID":"fb574051-68be-4a1a-8c3a-daae2a838fa2","Type":"ContainerStarted","Data":"66fe9276f320ef749d9c54598bac8d659b316338d0e772f2ce913d187b50ae99"}
Oct 30 22:16:11.084268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:11.084216    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:11.137493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:11.124441    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl" podStartSLOduration=2.1244104 podStartE2EDuration="2.1244104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:11.121931817 +0000 UTC m=+2189.780028044" watchObservedRunningTime="2026-10-30 22:16:11.1244104 +0000 UTC m=+2189.782506665"
Oct 30 22:16:11.430492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:11.424003    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:16:11.430492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:11.424059    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 22:16:12.419480 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:12.419444    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:16:12.419928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:12.419536    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 22:16:13.429493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:13.421963    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:16:13.429493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:13.422077    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 22:16:14.424641 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:14.424606    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:14.425019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:14.424999    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:14.425442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:14.425421    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:16:14.851919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:14.851816    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw" podUID="55a2c293-9e92-4573-a6c8-269466b6dc71" containerName="payload-processing" containerID="cri-o://52d806444492a84cc466e1bdc4c611e5a745a3daa44d0169c92f7d3500fde61b" gracePeriod=25
Oct 30 22:16:15.135601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.135442    4282 generic.go:356] "Generic (PLEG): container finished" podID="55a2c293-9e92-4573-a6c8-269466b6dc71" containerID="52d806444492a84cc466e1bdc4c611e5a745a3daa44d0169c92f7d3500fde61b" exitCode=0
Oct 30 22:16:15.136792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.136755    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw" event={"ID":"55a2c293-9e92-4573-a6c8-269466b6dc71","Type":"ContainerDied","Data":"52d806444492a84cc466e1bdc4c611e5a745a3daa44d0169c92f7d3500fde61b"}
Oct 30 22:16:15.236095 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.236066    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:16:15.347809 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.347778    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-provider-credentials\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-provider-credentials\") pod \"55a2c293-9e92-4573-a6c8-269466b6dc71\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") "
Oct 30 22:16:15.348012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.347996    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-plugins-config-volume\") pod \"55a2c293-9e92-4573-a6c8-269466b6dc71\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") "
Oct 30 22:16:15.348164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.348148    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-kube-api-access-pzzr8\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-kube-api-access-pzzr8\") pod \"55a2c293-9e92-4573-a6c8-269466b6dc71\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") "
Oct 30 22:16:15.348235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.348222    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-routing-overlay-volume\") pod \"55a2c293-9e92-4573-a6c8-269466b6dc71\" (UID: \"55a2c293-9e92-4573-a6c8-269466b6dc71\") "
Oct 30 22:16:15.348571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.348400    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-plugins-config-volume" pod "55a2c293-9e92-4573-a6c8-269466b6dc71" (UID: "55a2c293-9e92-4573-a6c8-269466b6dc71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:15.348721 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.348597    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-routing-overlay-volume" pod "55a2c293-9e92-4573-a6c8-269466b6dc71" (UID: "55a2c293-9e92-4573-a6c8-269466b6dc71"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:15.348818 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.348801    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:15.348859 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.348820    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55a2c293-9e92-4573-a6c8-269466b6dc71-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:15.350163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.350133    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-provider-credentials" pod "55a2c293-9e92-4573-a6c8-269466b6dc71" (UID: "55a2c293-9e92-4573-a6c8-269466b6dc71"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:15.350395 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.350375    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-kube-api-access-pzzr8" pod "55a2c293-9e92-4573-a6c8-269466b6dc71" (UID: "55a2c293-9e92-4573-a6c8-269466b6dc71"). InnerVolumeSpecName "kube-api-access-pzzr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:15.450221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.450195    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:15.450221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:15.450222    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzzr8\" (UniqueName: \"kubernetes.io/projected/55a2c293-9e92-4573-a6c8-269466b6dc71-kube-api-access-pzzr8\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:16.145313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:16.145278    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"
Oct 30 22:16:16.145875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:16.145314    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw" event={"ID":"55a2c293-9e92-4573-a6c8-269466b6dc71","Type":"ContainerDied","Data":"721626e9d756fb1ce78e05d279e646f9f272a6d13e4c47fa10e74c34def7c8cb"}
Oct 30 22:16:16.145875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:16.145377    4282 scope.go:122] "RemoveContainer" containerID="52d806444492a84cc466e1bdc4c611e5a745a3daa44d0169c92f7d3500fde61b"
Oct 30 22:16:16.165454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:16.165427    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"]
Oct 30 22:16:16.174529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:16.174505    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-ccbd89qkw"]
Oct 30 22:16:17.467531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:17.467500    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a2c293-9e92-4573-a6c8-269466b6dc71" path="/var/lib/kubelet/pods/55a2c293-9e92-4573-a6c8-269466b6dc71/volumes"
Oct 30 22:16:22.109976 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:22.109948    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:16:34.650885 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:16:34.650851    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:16:45.768684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:45.768653    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"]
Oct 30 22:16:48.014914 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:48.014883    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:16:50.003882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.003847    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"]
Oct 30 22:16:50.004876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.004847    4282 cpu_manager.go:414] "containerMap: removing container" podUID="55a2c293-9e92-4573-a6c8-269466b6dc71" containerName="payload-processing"
Oct 30 22:16:50.004876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.004868    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a2c293-9e92-4573-a6c8-269466b6dc71" containerName="payload-processing"
Oct 30 22:16:50.005071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.005057    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a2c293-9e92-4573-a6c8-269466b6dc71" containerName="payload-processing"
Oct 30 22:16:50.022903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.022860    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.024704 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.024674    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"]
Oct 30 22:16:50.073595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.073564    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.073792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.073633    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.073792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.073783    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.073928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.073826    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.073928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.073920    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.074053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.073943    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnvdj\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-kube-api-access-jnvdj\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.074053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.074003    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.074167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.074052    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.074298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.074201    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.175906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.175875    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.175942    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.175998    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176063 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176030    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176067    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176097    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnvdj\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-kube-api-access-jnvdj\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176128    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176376    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176512    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176579    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176865    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.176936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176892    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.177105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.176971    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.177105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.177037    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.178351 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.178329    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.178705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.178685    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.187650 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.187629    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnvdj\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-kube-api-access-jnvdj\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.187913 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.187893    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.342708 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.342615    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:50.648030 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.648001    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"]
Oct 30 22:16:50.648406 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:16:50.648384    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9cd784_b387_42ba_b0d3_c0a3c2dadc7d.slice/crio-33ace5388edd6545fb4e7b3166edd3cdfea00711e5669da897a62aced046bb90 WatchSource:0}: Error finding container 33ace5388edd6545fb4e7b3166edd3cdfea00711e5669da897a62aced046bb90: Status 404 returned error can't find the container with id 33ace5388edd6545fb4e7b3166edd3cdfea00711e5669da897a62aced046bb90
Oct 30 22:16:50.651854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.651816    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:16:50.651959 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.651891    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:16:50.651959 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.651923    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:16:50.776360 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.771890    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97" podUID="59cf758c-b4bb-4797-bcec-113d8262c26f" containerName="payload-processing" containerID="cri-o://de2069638a430664f643c8a5e1672d95f7a644afe24850c8c77cf7486984a3ff" gracePeriod=25
Oct 30 22:16:50.936364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.936333    4282 generic.go:356] "Generic (PLEG): container finished" podID="59cf758c-b4bb-4797-bcec-113d8262c26f" containerID="de2069638a430664f643c8a5e1672d95f7a644afe24850c8c77cf7486984a3ff" exitCode=0
Oct 30 22:16:50.936596 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.936414    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97" event={"ID":"59cf758c-b4bb-4797-bcec-113d8262c26f","Type":"ContainerDied","Data":"de2069638a430664f643c8a5e1672d95f7a644afe24850c8c77cf7486984a3ff"}
Oct 30 22:16:50.938427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.938393    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" event={"ID":"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d","Type":"ContainerStarted","Data":"ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a"}
Oct 30 22:16:50.938585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.938451    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" event={"ID":"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d","Type":"ContainerStarted","Data":"33ace5388edd6545fb4e7b3166edd3cdfea00711e5669da897a62aced046bb90"}
Oct 30 22:16:50.969312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:50.969251    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" podStartSLOduration=1.969237401 podStartE2EDuration="1.969237401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:50.961985299 +0000 UTC m=+2229.620081523" watchObservedRunningTime="2026-10-30 22:16:50.969237401 +0000 UTC m=+2229.627333624"
Oct 30 22:16:51.090408 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.090383    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:16:51.096961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.096940    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-kube-api-access-74xk4\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-kube-api-access-74xk4\") pod \"59cf758c-b4bb-4797-bcec-113d8262c26f\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") "
Oct 30 22:16:51.097127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.097116    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-plugins-config-volume\") pod \"59cf758c-b4bb-4797-bcec-113d8262c26f\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") "
Oct 30 22:16:51.097179 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.097161    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-routing-overlay-volume\") pod \"59cf758c-b4bb-4797-bcec-113d8262c26f\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") "
Oct 30 22:16:51.097224 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.097196    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-provider-credentials\") pod \"59cf758c-b4bb-4797-bcec-113d8262c26f\" (UID: \"59cf758c-b4bb-4797-bcec-113d8262c26f\") "
Oct 30 22:16:51.097588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.097549    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-plugins-config-volume" pod "59cf758c-b4bb-4797-bcec-113d8262c26f" (UID: "59cf758c-b4bb-4797-bcec-113d8262c26f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:51.097588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.097575    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-routing-overlay-volume" pod "59cf758c-b4bb-4797-bcec-113d8262c26f" (UID: "59cf758c-b4bb-4797-bcec-113d8262c26f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:51.098245 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.098223    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:51.098298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.098255    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59cf758c-b4bb-4797-bcec-113d8262c26f-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:51.099264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.099241    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-kube-api-access-74xk4" pod "59cf758c-b4bb-4797-bcec-113d8262c26f" (UID: "59cf758c-b4bb-4797-bcec-113d8262c26f"). InnerVolumeSpecName "kube-api-access-74xk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:51.099417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.099400    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-provider-credentials" pod "59cf758c-b4bb-4797-bcec-113d8262c26f" (UID: "59cf758c-b4bb-4797-bcec-113d8262c26f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:51.200509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.199934    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74xk4\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-kube-api-access-74xk4\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:51.200509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.199974    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59cf758c-b4bb-4797-bcec-113d8262c26f-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:51.342954 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.342920    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:51.345078 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.345052    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 22:16:51.345234 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.345110    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 22:16:51.947382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.947343    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97" event={"ID":"59cf758c-b4bb-4797-bcec-113d8262c26f","Type":"ContainerDied","Data":"dd0e271c2005bfd141c2919c0e82204b02f2774f57ac6e2ea3443aa8ac6e2b2d"}
Oct 30 22:16:51.947382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.947378    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"
Oct 30 22:16:51.947667 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.947432    4282 scope.go:122] "RemoveContainer" containerID="de2069638a430664f643c8a5e1672d95f7a644afe24850c8c77cf7486984a3ff"
Oct 30 22:16:51.989387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.989354    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"]
Oct 30 22:16:51.993905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:51.993878    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-28b4bb/payload-processing-external-model-e2e-extmod-b-28b4bb-5575npj97"]
Oct 30 22:16:52.343304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:52.343226    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 22:16:52.343304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:52.343280    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 22:16:53.369497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:53.356570    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 22:16:53.369497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:53.356636    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 22:16:53.482392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:53.482358    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59cf758c-b4bb-4797-bcec-113d8262c26f" path="/var/lib/kubelet/pods/59cf758c-b4bb-4797-bcec-113d8262c26f/volumes"
Oct 30 22:16:54.343525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:54.343496    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 22:16:54.343720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:54.343549    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 22:16:55.343234 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:55.343203    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 22:16:55.343693 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:55.343259    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 22:16:56.343556 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:56.343519    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused" start-of-body=
Oct 30 22:16:56.343970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:56.343581    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.236:15021/healthz/ready\": dial tcp 10.133.0.236:15021: connect: connection refused"
Oct 30 22:16:57.380907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:57.380825    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:57.382327 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:57.382300    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:16:57.383329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:16:57.383310    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:17:04.651162 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:17:04.651127    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:17:34.651672 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:17:34.651630    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:17:36.056122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:36.056084    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"]
Oct 30 22:17:39.758305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:39.758273    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:17:41.056804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.056765    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl" podUID="fb574051-68be-4a1a-8c3a-daae2a838fa2" containerName="payload-processing" containerID="cri-o://5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3" gracePeriod=25
Oct 30 22:17:41.353557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.353530    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:17:41.361131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.361108    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-provider-credentials\") pod \"fb574051-68be-4a1a-8c3a-daae2a838fa2\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") "
Oct 30 22:17:41.361272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.361197    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-routing-overlay-volume\") pod \"fb574051-68be-4a1a-8c3a-daae2a838fa2\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") "
Oct 30 22:17:41.361354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.361338    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-kube-api-access-5mpxt\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-kube-api-access-5mpxt\") pod \"fb574051-68be-4a1a-8c3a-daae2a838fa2\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") "
Oct 30 22:17:41.361456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.361433    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-plugins-config-volume\") pod \"fb574051-68be-4a1a-8c3a-daae2a838fa2\" (UID: \"fb574051-68be-4a1a-8c3a-daae2a838fa2\") "
Oct 30 22:17:41.361693 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.361603    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-routing-overlay-volume" pod "fb574051-68be-4a1a-8c3a-daae2a838fa2" (UID: "fb574051-68be-4a1a-8c3a-daae2a838fa2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:41.361968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.361822    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-plugins-config-volume" pod "fb574051-68be-4a1a-8c3a-daae2a838fa2" (UID: "fb574051-68be-4a1a-8c3a-daae2a838fa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:41.362130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.362108    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.362207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.362131    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb574051-68be-4a1a-8c3a-daae2a838fa2-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.363409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.363382    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-provider-credentials" pod "fb574051-68be-4a1a-8c3a-daae2a838fa2" (UID: "fb574051-68be-4a1a-8c3a-daae2a838fa2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:41.363575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.363539    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-kube-api-access-5mpxt" pod "fb574051-68be-4a1a-8c3a-daae2a838fa2" (UID: "fb574051-68be-4a1a-8c3a-daae2a838fa2"). InnerVolumeSpecName "kube-api-access-5mpxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:41.463150 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.463116    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.463150 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.463143    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mpxt\" (UniqueName: \"kubernetes.io/projected/fb574051-68be-4a1a-8c3a-daae2a838fa2-kube-api-access-5mpxt\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:41.692520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.692451    4282 generic.go:356] "Generic (PLEG): container finished" podID="fb574051-68be-4a1a-8c3a-daae2a838fa2" containerID="5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3" exitCode=0
Oct 30 22:17:41.692731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.692632    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl" event={"ID":"fb574051-68be-4a1a-8c3a-daae2a838fa2","Type":"ContainerDied","Data":"5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3"}
Oct 30 22:17:41.692731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.692682    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl" event={"ID":"fb574051-68be-4a1a-8c3a-daae2a838fa2","Type":"ContainerDied","Data":"66fe9276f320ef749d9c54598bac8d659b316338d0e772f2ce913d187b50ae99"}
Oct 30 22:17:41.692731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.692709    4282 scope.go:122] "RemoveContainer" containerID="5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3"
Oct 30 22:17:41.692906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.692854    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"
Oct 30 22:17:41.706805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.706782    4282 scope.go:122] "RemoveContainer" containerID="5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3"
Oct 30 22:17:41.707119 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:17:41.707097    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3\": container with ID starting with 5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3 not found: ID does not exist" containerID="5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3"
Oct 30 22:17:41.707168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.707129    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3"} err="failed to get container status \"5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3\": rpc error: code = NotFound desc = could not find container \"5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3\": container with ID starting with 5f41e3e0dd1564267e741f65b9e83ab388e40264fee142951827991bf04b33d3 not found: ID does not exist"
Oct 30 22:17:41.712731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.712571    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"]
Oct 30 22:17:41.717202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:41.717177    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bacea0/payload-processing-external-model-e2e-extmod-a-bacea0-d9469dxcl"]
Oct 30 22:17:43.526492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:17:43.521907    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb574051-68be-4a1a-8c3a-daae2a838fa2" path="/var/lib/kubelet/pods/fb574051-68be-4a1a-8c3a-daae2a838fa2/volumes"
Oct 30 22:18:01.008552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:00.949595    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"]
Oct 30 22:18:04.681914 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:18:04.681865    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:18:06.030407 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.030366    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr" podUID="ab3b25d4-5b07-4c09-9670-a772432bd313" containerName="payload-processing" containerID="cri-o://9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73" gracePeriod=25
Oct 30 22:18:06.333100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.333062    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:18:06.420360 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.420327    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-provider-credentials\") pod \"ab3b25d4-5b07-4c09-9670-a772432bd313\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") "
Oct 30 22:18:06.420579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.420386    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-plugins-config-volume\") pod \"ab3b25d4-5b07-4c09-9670-a772432bd313\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") "
Oct 30 22:18:06.420669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.420650    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-kube-api-access-8m7hc\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-kube-api-access-8m7hc\") pod \"ab3b25d4-5b07-4c09-9670-a772432bd313\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") "
Oct 30 22:18:06.420794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.420777    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-routing-overlay-volume\") pod \"ab3b25d4-5b07-4c09-9670-a772432bd313\" (UID: \"ab3b25d4-5b07-4c09-9670-a772432bd313\") "
Oct 30 22:18:06.420864 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.420796    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-plugins-config-volume" pod "ab3b25d4-5b07-4c09-9670-a772432bd313" (UID: "ab3b25d4-5b07-4c09-9670-a772432bd313"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:06.421148 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.421093    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-routing-overlay-volume" pod "ab3b25d4-5b07-4c09-9670-a772432bd313" (UID: "ab3b25d4-5b07-4c09-9670-a772432bd313"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:06.421419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.421402    4282 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-routing-overlay-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:06.421529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.421420    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3b25d4-5b07-4c09-9670-a772432bd313-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:06.422842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.422821    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-provider-credentials" pod "ab3b25d4-5b07-4c09-9670-a772432bd313" (UID: "ab3b25d4-5b07-4c09-9670-a772432bd313"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:06.422940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.422923    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-kube-api-access-8m7hc" pod "ab3b25d4-5b07-4c09-9670-a772432bd313" (UID: "ab3b25d4-5b07-4c09-9670-a772432bd313"). InnerVolumeSpecName "kube-api-access-8m7hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:06.523153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.523123    4282 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-provider-credentials\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:06.523153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.523144    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m7hc\" (UniqueName: \"kubernetes.io/projected/ab3b25d4-5b07-4c09-9670-a772432bd313-kube-api-access-8m7hc\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:06.723110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.721579    4282 generic.go:356] "Generic (PLEG): container finished" podID="ab3b25d4-5b07-4c09-9670-a772432bd313" containerID="9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73" exitCode=0
Oct 30 22:18:06.723110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.721623    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr" event={"ID":"ab3b25d4-5b07-4c09-9670-a772432bd313","Type":"ContainerDied","Data":"9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73"}
Oct 30 22:18:06.723110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.721707    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr" event={"ID":"ab3b25d4-5b07-4c09-9670-a772432bd313","Type":"ContainerDied","Data":"a78cd7a8e4e661e8066eedd4c4d5771355227f0cafe5d186749ce0a094e78b76"}
Oct 30 22:18:06.723110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.721748    4282 scope.go:122] "RemoveContainer" containerID="9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73"
Oct 30 22:18:06.723110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.721651    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"
Oct 30 22:18:06.744541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.744446    4282 scope.go:122] "RemoveContainer" containerID="9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73"
Oct 30 22:18:06.744780 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:18:06.744759    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73\": container with ID starting with 9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73 not found: ID does not exist" containerID="9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73"
Oct 30 22:18:06.744835 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.744788    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73"} err="failed to get container status \"9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73\": rpc error: code = NotFound desc = could not find container \"9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73\": container with ID starting with 9aecfcc9db0ce111f827e8f0b4d2cb100ddf47c2a33dc4e8c65b1a5f14831c73 not found: ID does not exist"
Oct 30 22:18:06.765904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.765878    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"]
Oct 30 22:18:06.770176 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:06.770150    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6f8222/payload-processing-external-model-e2e-extmod-a-6f8222-59899x4cr"]
Oct 30 22:18:07.540493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:07.534040    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3b25d4-5b07-4c09-9670-a772432bd313" path="/var/lib/kubelet/pods/ab3b25d4-5b07-4c09-9670-a772432bd313/volumes"
Oct 30 22:18:20.362493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:20.361631    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-c5n7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:18:20.362493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:20.361720    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-c5n7j" podUID="f536d905-2529-4495-80ff-1d16c5e2cc8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:27.470555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.470524    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"]
Oct 30 22:18:27.471583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471562    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ab3b25d4-5b07-4c09-9670-a772432bd313" containerName="payload-processing"
Oct 30 22:18:27.471583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471584    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3b25d4-5b07-4c09-9670-a772432bd313" containerName="payload-processing"
Oct 30 22:18:27.471775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471630    4282 cpu_manager.go:414] "containerMap: removing container" podUID="59cf758c-b4bb-4797-bcec-113d8262c26f" containerName="payload-processing"
Oct 30 22:18:27.471775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471639    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59cf758c-b4bb-4797-bcec-113d8262c26f" containerName="payload-processing"
Oct 30 22:18:27.471775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471680    4282 cpu_manager.go:414] "containerMap: removing container" podUID="fb574051-68be-4a1a-8c3a-daae2a838fa2" containerName="payload-processing"
Oct 30 22:18:27.471775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471687    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb574051-68be-4a1a-8c3a-daae2a838fa2" containerName="payload-processing"
Oct 30 22:18:27.471986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471878    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="59cf758c-b4bb-4797-bcec-113d8262c26f" containerName="payload-processing"
Oct 30 22:18:27.471986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471900    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb574051-68be-4a1a-8c3a-daae2a838fa2" containerName="payload-processing"
Oct 30 22:18:27.471986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.471917    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3b25d4-5b07-4c09-9670-a772432bd313" containerName="payload-processing"
Oct 30 22:18:27.476255 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.476222    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.496945 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.496923    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"]
Oct 30 22:18:27.504836 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.504801    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.505015 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.504853    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54455775-f1d8-4e14-afe1-fd8ad99fb188-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.505015 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.504928    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.505015 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.504989    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.505204 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.505094    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.505266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.505248    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.505383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.505361    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.505445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.505419    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wj4\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-kube-api-access-97wj4\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.505558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.505532    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607304    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607364    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wj4\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-kube-api-access-97wj4\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607412    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607551    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607583    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54455775-f1d8-4e14-afe1-fd8ad99fb188-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607678    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607720    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607792    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.607922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607828    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.608041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607930    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.608041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607952    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.608158 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.607931    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.608211 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.608188    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.608368 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.608347    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54455775-f1d8-4e14-afe1-fd8ad99fb188-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.610406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.610376    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.610540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.610509    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.620304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.620283    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.620778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.620755    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wj4\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-kube-api-access-97wj4\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gx9nl\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:27.795667 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:27.795560    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:28.127761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:28.127458    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"]
Oct 30 22:18:28.136254 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:28.136214    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:18:28.136397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:28.136307    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:18:28.136397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:28.136354    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:18:29.197074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:29.197037    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" event={"ID":"54455775-f1d8-4e14-afe1-fd8ad99fb188","Type":"ContainerStarted","Data":"ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff"}
Oct 30 22:18:29.197750 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:29.197707    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" event={"ID":"54455775-f1d8-4e14-afe1-fd8ad99fb188","Type":"ContainerStarted","Data":"6b6aa100e426d12b8a023a338ef047801033c61dc01a3b226ad208e982f6cbef"}
Oct 30 22:18:29.811493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:29.796800    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:29.811493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:29.803384    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gx9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:18:29.811493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:29.803440    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:18:30.795975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:30.795938    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gx9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:18:30.796486 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:30.796034    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:18:31.796501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:31.796425    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gx9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:18:31.796909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:31.796542    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:18:32.796083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:32.796047    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gx9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:18:32.796278 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:32.796100    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:18:33.796387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:33.796349    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gx9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:18:33.796962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:33.796924    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:18:34.650648 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:18:34.650607    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:18:34.796786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:34.796751    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gx9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:18:34.797285 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:34.796813    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:18:35.800640 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:35.800598    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gx9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:18:35.801181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:35.800695    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:18:36.796318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:36.796279    4282 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gx9nl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Oct 30 22:18:36.796565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:36.796348    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Oct 30 22:18:37.854771 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:37.854741    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:37.855300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:37.855160    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:37.855910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:37.855892    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:18:37.881928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:37.881879    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podStartSLOduration=10.881864138 podStartE2EDuration="10.881864138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:29.232599053 +0000 UTC m=+2327.890695278" watchObservedRunningTime="2026-10-30 22:18:37.881864138 +0000 UTC m=+2336.539960425"
Oct 30 22:18:53.897183 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:18:53.897147    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:19:04.650754 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:19:04.650714    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:19:27.102334 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:27.102249    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:19:34.651248 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:19:34.651213    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:19:43.868808 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:43.868770    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"]
Oct 30 22:19:43.869484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:43.869041    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy" containerID="cri-o://b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee" gracePeriod=30
Oct 30 22:19:49.184694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.184667    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:19:49.254634 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.254601    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-credential-socket\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.254824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.254688    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-envoy\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.254824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.254771    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-certs\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.254988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.254917    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-data\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.254988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.254949    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-credential-socket" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:49.254988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.254984    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-kube-api-access-l27jt\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-kube-api-access-l27jt\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.255159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.255048    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-podinfo\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.255159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.255128    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-token\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-token\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.255271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.255173    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istiod-ca-cert\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.255271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.255215    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-socket\") pod \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\" (UID: \"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e\") "
Oct 30 22:19:49.256589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.255721    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-data" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:49.256589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.256297    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.256589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.256303    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istiod-ca-cert" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:49.256589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.256402    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-socket" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:49.257339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.257315    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-certs" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:49.257779 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.257750    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-envoy" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:49.258244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.258220    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-token" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:49.258601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.256320    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.258810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.258785    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-kube-api-access-l27jt" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "kube-api-access-l27jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:49.259907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.259876    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-podinfo" pod "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" (UID: "aded6eba-1991-4d9d-8e01-5cc2d7a0c74e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:49.365512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.365338    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.365512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.365378    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.365512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.365393    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l27jt\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-kube-api-access-l27jt\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.365512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.365410    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.365512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.365424    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.365512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.365437    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.365512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.365450    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:49.730928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.730842    4282 generic.go:356] "Generic (PLEG): container finished" podID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerID="b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee" exitCode=0
Oct 30 22:19:49.730928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.730906    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"
Oct 30 22:19:49.731127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.730925    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" event={"ID":"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e","Type":"ContainerDied","Data":"b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee"}
Oct 30 22:19:49.731127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.730982    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb" event={"ID":"aded6eba-1991-4d9d-8e01-5cc2d7a0c74e","Type":"ContainerDied","Data":"0abb8aa2872b276334aa5b1a480669a3d27d269267819ca8b3ae895e22cc4228"}
Oct 30 22:19:49.731127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.731006    4282 scope.go:122] "RemoveContainer" containerID="b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee"
Oct 30 22:19:49.742636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.742615    4282 scope.go:122] "RemoveContainer" containerID="b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee"
Oct 30 22:19:49.742911 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:19:49.742889    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee\": container with ID starting with b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee not found: ID does not exist" containerID="b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee"
Oct 30 22:19:49.742981 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.742922    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee"} err="failed to get container status \"b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee\": rpc error: code = NotFound desc = could not find container \"b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee\": container with ID starting with b6e6dffb5631671ef4647ee158dbbb3c7da06c5e5f41f35dd17d4868ce5b6cee not found: ID does not exist"
Oct 30 22:19:49.750424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.750402    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"]
Oct 30 22:19:49.756804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:49.756784    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-78kcb"]
Oct 30 22:19:51.468894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:51.468863    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" path="/var/lib/kubelet/pods/aded6eba-1991-4d9d-8e01-5cc2d7a0c74e/volumes"
Oct 30 22:19:58.209849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:19:58.209816    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:20:04.652193 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:20:04.652145    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:20:24.027864 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:20:24.027828    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"]
Oct 30 22:20:24.032087 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:20:24.032061    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-536e9795-4475f"]
Oct 30 22:20:25.466887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:20:25.466849    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95212ca9-d770-4bb4-9586-0509688b960d" path="/var/lib/kubelet/pods/95212ca9-d770-4bb4-9586-0509688b960d/volumes"
Oct 30 22:20:32.988990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:20:32.988953    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:20:34.650727 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:20:34.650671    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:21:04.909140 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:21:04.909092    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:21:06.310886 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:21:06.310855    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:21:34.650794 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:21:34.650743    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:21:45.399127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:21:45.399093    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:21:45.646938 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:21:45.646906    4282 scope.go:122] "RemoveContainer" containerID="ff40b2e3a2690a10ba9fdb606b50cc5682cfd5e6916633a722bdb5045d8a8dc5"
Oct 30 22:22:04.651634 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:22:04.651595    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:22:05.538647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:05.538614    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"]
Oct 30 22:22:05.538848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:05.538824    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy" containerID="cri-o://b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d" gracePeriod=30
Oct 30 22:22:06.675807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:06.675317    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:22:06.707018 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:06.706927    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:22:10.843532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.843442    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:22:10.979523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.979365    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-kube-api-access-dmqxv\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-kube-api-access-dmqxv\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.981994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.981966    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/579717b1-feef-4e2a-9da8-62721859adeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/579717b1-feef-4e2a-9da8-62721859adeb-istiod-ca-cert\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.982533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.982516    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-socket\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.983133 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.983103    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/579717b1-feef-4e2a-9da8-62721859adeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/579717b1-feef-4e2a-9da8-62721859adeb-istio-podinfo\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.983932 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.983912    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-data\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.986769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.986750    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-certs\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.987007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.986940    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-istio-token\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-istio-token\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.987308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.987289    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-credential-socket\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.987520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.987441    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-envoy\") pod \"579717b1-feef-4e2a-9da8-62721859adeb\" (UID: \"579717b1-feef-4e2a-9da8-62721859adeb\") "
Oct 30 22:22:10.989270 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.983317    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-socket" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:10.989448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.983756    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579717b1-feef-4e2a-9da8-62721859adeb-istiod-ca-cert" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:10.989619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.986297    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-data" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:10.989707 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.989435    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-credential-socket" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:10.990245 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.990223    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-certs" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:10.991884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.991856    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-istio-token" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:10.991998 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.991874    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-kube-api-access-dmqxv" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "kube-api-access-dmqxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:10.992110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.992080    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/579717b1-feef-4e2a-9da8-62721859adeb-istio-podinfo" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:22:10.992538 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:10.992518    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-envoy" pod "579717b1-feef-4e2a-9da8-62721859adeb" (UID: "579717b1-feef-4e2a-9da8-62721859adeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:22:11.010451 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.010412    4282 generic.go:356] "Generic (PLEG): container finished" podID="579717b1-feef-4e2a-9da8-62721859adeb" containerID="b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d" exitCode=0
Oct 30 22:22:11.010642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.010496    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" event={"ID":"579717b1-feef-4e2a-9da8-62721859adeb","Type":"ContainerDied","Data":"b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d"}
Oct 30 22:22:11.010642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.010515    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"
Oct 30 22:22:11.010642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.010580    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq" event={"ID":"579717b1-feef-4e2a-9da8-62721859adeb","Type":"ContainerDied","Data":"fb169fcff70dc9a7b1971e5f10d94383ccca9bccd497eb98c3cf124c51709057"}
Oct 30 22:22:11.010642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.010604    4282 scope.go:122] "RemoveContainer" containerID="b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d"
Oct 30 22:22:11.028600 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.028583    4282 scope.go:122] "RemoveContainer" containerID="8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d"
Oct 30 22:22:11.038704 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.038682    4282 scope.go:122] "RemoveContainer" containerID="b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d"
Oct 30 22:22:11.039007 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:22:11.038979    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d\": container with ID starting with b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d not found: ID does not exist" containerID="b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d"
Oct 30 22:22:11.039086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.039008    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d"} err="failed to get container status \"b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d\": rpc error: code = NotFound desc = could not find container \"b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d\": container with ID starting with b5322c5183975fe34a7aeeccbc34c36f412ef584cb479222091f1165b321da8d not found: ID does not exist"
Oct 30 22:22:11.039086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.039029    4282 scope.go:122] "RemoveContainer" containerID="8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d"
Oct 30 22:22:11.039389 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:22:11.039360    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d\": container with ID starting with 8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d not found: ID does not exist" containerID="8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d"
Oct 30 22:22:11.039530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.039397    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d"} err="failed to get container status \"8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d\": rpc error: code = NotFound desc = could not find container \"8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d\": container with ID starting with 8df4d005bb49f419dbf7a1b1215711ba02f7657c022caa778cf7ab54cef2d28d not found: ID does not exist"
Oct 30 22:22:11.041304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.041280    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"]
Oct 30 22:22:11.045646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.045624    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-k68wq"]
Oct 30 22:22:11.089924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089899    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.090010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089924    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.090010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089940    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.090010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089952    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmqxv\" (UniqueName: \"kubernetes.io/projected/579717b1-feef-4e2a-9da8-62721859adeb-kube-api-access-dmqxv\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.090010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089966    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/579717b1-feef-4e2a-9da8-62721859adeb-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.090010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089975    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.090010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089983    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/579717b1-feef-4e2a-9da8-62721859adeb-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.090010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089991    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.090010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.089999    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/579717b1-feef-4e2a-9da8-62721859adeb-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:22:11.466127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:11.466092    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579717b1-feef-4e2a-9da8-62721859adeb" path="/var/lib/kubelet/pods/579717b1-feef-4e2a-9da8-62721859adeb/volumes"
Oct 30 22:22:33.442310 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:22:33.442277    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:22:34.786429 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:22:34.786388    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:23:04.652238 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:23:04.652203    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:23:10.851308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:10.851275    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"]
Oct 30 22:23:10.851813 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:10.851518    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy" containerID="cri-o://3768f4a2356dd3a2b4ffa71a332efc5e934639992e590411fe464026ddcf16a4" gracePeriod=30
Oct 30 22:23:12.553156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:12.553122    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"]
Oct 30 22:23:12.553607 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:12.553331    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy" containerID="cri-o://733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f" gracePeriod=30
Oct 30 22:23:15.983892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:15.983859    4282 generic.go:356] "Generic (PLEG): container finished" podID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerID="3768f4a2356dd3a2b4ffa71a332efc5e934639992e590411fe464026ddcf16a4" exitCode=0
Oct 30 22:23:15.984391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:15.983928    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" event={"ID":"e348f2cc-4f24-47ad-b0c6-41d667b29f78","Type":"ContainerDied","Data":"3768f4a2356dd3a2b4ffa71a332efc5e934639992e590411fe464026ddcf16a4"}
Oct 30 22:23:16.136997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.136931    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 22:23:16.302500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302052    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-certs\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.302500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302178    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istiod-ca-cert\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.302500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302260    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-token\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-token\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.302500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302323    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-envoy\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.302500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302379    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-data\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.302975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302666    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-credential-socket\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.302975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302697    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-kube-api-access-xzhhq\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-kube-api-access-xzhhq\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.302975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302808    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-socket\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.302975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.302861    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-podinfo\") pod \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\" (UID: \"e348f2cc-4f24-47ad-b0c6-41d667b29f78\") "
Oct 30 22:23:16.315498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.315443    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-data" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:16.332669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.317232    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-credential-socket" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:16.332669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.317333    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-podinfo" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:16.332669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.327760    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istiod-ca-cert" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:16.332669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.332577    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-socket" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:16.333306 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.333277    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-certs" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:16.347450 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.347385    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-envoy" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:16.347962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.347922    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-token" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:16.348302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.348275    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-kube-api-access-xzhhq" pod "e348f2cc-4f24-47ad-b0c6-41d667b29f78" (UID: "e348f2cc-4f24-47ad-b0c6-41d667b29f78"). InnerVolumeSpecName "kube-api-access-xzhhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:16.405883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405805    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.405883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405836    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.405883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405845    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.405883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405854    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.405883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405864    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.405883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405873    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.405883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405881    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.405883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405889    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e348f2cc-4f24-47ad-b0c6-41d667b29f78-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:16.406253 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:16.405898    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzhhq\" (UniqueName: \"kubernetes.io/projected/e348f2cc-4f24-47ad-b0c6-41d667b29f78-kube-api-access-xzhhq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:17.009940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:17.009904    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx" event={"ID":"e348f2cc-4f24-47ad-b0c6-41d667b29f78","Type":"ContainerDied","Data":"dc272a7ff58a6c03d0b86cc6a204f96eeab78bfd076d86fab423ff6884e2dd00"}
Oct 30 22:23:17.009940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:17.009927    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"
Oct 30 22:23:17.010549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:17.009967    4282 scope.go:122] "RemoveContainer" containerID="3768f4a2356dd3a2b4ffa71a332efc5e934639992e590411fe464026ddcf16a4"
Oct 30 22:23:17.043970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:17.043938    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"]
Oct 30 22:23:17.048680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:17.048655    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-bzblx"]
Oct 30 22:23:17.466302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:17.466268    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" path="/var/lib/kubelet/pods/e348f2cc-4f24-47ad-b0c6-41d667b29f78/volumes"
Oct 30 22:23:17.904830 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:17.904797    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 22:23:18.019262 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.019228    4282 generic.go:356] "Generic (PLEG): container finished" podID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerID="733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f" exitCode=0
Oct 30 22:23:18.019797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.019286    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"
Oct 30 22:23:18.019797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.019319    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" event={"ID":"8a025f32-89c7-4009-8b77-7ee839ed8c20","Type":"ContainerDied","Data":"733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f"}
Oct 30 22:23:18.019797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.019374    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj" event={"ID":"8a025f32-89c7-4009-8b77-7ee839ed8c20","Type":"ContainerDied","Data":"2e7be08d2995a5ce26b50c6740e6c7c0f7a187e649bc9f71b6ccf83f32aa760e"}
Oct 30 22:23:18.019797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.019406    4282 scope.go:122] "RemoveContainer" containerID="733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f"
Oct 30 22:23:18.030365 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.030349    4282 scope.go:122] "RemoveContainer" containerID="f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec"
Oct 30 22:23:18.039445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.039366    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-socket\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.040017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.039901    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-socket" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:18.040225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.040106    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a025f32-89c7-4009-8b77-7ee839ed8c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a025f32-89c7-4009-8b77-7ee839ed8c20-istiod-ca-cert\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.040225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.040156    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-podinfo\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.040446 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.040271    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-credential-socket\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.041129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.040889    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-credential-socket" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:18.041129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.041094    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-data" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:18.041317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.041145    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a025f32-89c7-4009-8b77-7ee839ed8c20-istiod-ca-cert" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:18.041551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.041443    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-data\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.042118 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.042094    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-certs" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:18.042243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.042133    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-certs\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.042243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.042185    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-kube-api-access-lhcks\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-kube-api-access-lhcks\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.042398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.042285    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-token\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-token\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.042613 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.042316    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-envoy\") pod \"8a025f32-89c7-4009-8b77-7ee839ed8c20\" (UID: \"8a025f32-89c7-4009-8b77-7ee839ed8c20\") "
Oct 30 22:23:18.044046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.043959    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.044046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.043988    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a025f32-89c7-4009-8b77-7ee839ed8c20-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.044046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.044005    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.044046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.044018    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.044046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.044034    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.045810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.045653    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-kube-api-access-lhcks" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "kube-api-access-lhcks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:18.045946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.045417    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-podinfo" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:18.047991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.047953    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-envoy" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:18.048214 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.048191    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-token" pod "8a025f32-89c7-4009-8b77-7ee839ed8c20" (UID: "8a025f32-89c7-4009-8b77-7ee839ed8c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:18.050367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.050207    4282 scope.go:122] "RemoveContainer" containerID="733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f"
Oct 30 22:23:18.051156 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:23:18.050901    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f\": container with ID starting with 733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f not found: ID does not exist" containerID="733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f"
Oct 30 22:23:18.051156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.050940    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f"} err="failed to get container status \"733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f\": rpc error: code = NotFound desc = could not find container \"733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f\": container with ID starting with 733916311e4811f6577dc643a84c76aa6a1ee02298ebd245266e8f2fee93b64f not found: ID does not exist"
Oct 30 22:23:18.051156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.051043    4282 scope.go:122] "RemoveContainer" containerID="f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec"
Oct 30 22:23:18.052840 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:23:18.052814    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec\": container with ID starting with f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec not found: ID does not exist" containerID="f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec"
Oct 30 22:23:18.052932 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.052846    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec"} err="failed to get container status \"f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec\": rpc error: code = NotFound desc = could not find container \"f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec\": container with ID starting with f48b553e82ad0179e1f8d18f25a2c35cba9cadfe1ad112c9d6b86a3a71cfe6ec not found: ID does not exist"
Oct 30 22:23:18.145420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.145382    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.145420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.145414    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.145630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.145430    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a025f32-89c7-4009-8b77-7ee839ed8c20-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.145630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.145443    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhcks\" (UniqueName: \"kubernetes.io/projected/8a025f32-89c7-4009-8b77-7ee839ed8c20-kube-api-access-lhcks\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:18.412548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.409920    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"]
Oct 30 22:23:18.421526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:18.417783    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-kk6jj"]
Oct 30 22:23:19.466542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:19.466511    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" path="/var/lib/kubelet/pods/8a025f32-89c7-4009-8b77-7ee839ed8c20/volumes"
Oct 30 22:23:20.497918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:20.497885    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:23:25.473228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:25.473194    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"]
Oct 30 22:23:25.473717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:25.473414    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy" containerID="cri-o://09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119" gracePeriod=30
Oct 30 22:23:30.763146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.763085    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 22:23:30.812596 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.812560    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73d8a943-a45d-434a-a400-eec464164ea2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d8a943-a45d-434a-a400-eec464164ea2-istio-podinfo\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.812797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.812656    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-credential-socket\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.812875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.812803    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-certs\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.812875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.812846    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-istio-token\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-istio-token\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.812996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.812925    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-envoy\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.813067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.813041    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-socket\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.813162 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.813140    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-data\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.813242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.813186    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-kube-api-access-cr2fv\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-kube-api-access-cr2fv\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.813242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.813220    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d8a943-a45d-434a-a400-eec464164ea2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d8a943-a45d-434a-a400-eec464164ea2-istiod-ca-cert\") pod \"73d8a943-a45d-434a-a400-eec464164ea2\" (UID: \"73d8a943-a45d-434a-a400-eec464164ea2\") "
Oct 30 22:23:30.814252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.813410    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-socket" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.814252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.813765    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-data" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.814252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.814079    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-certs" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.814252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.814199    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.814252 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.814223    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.814630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.814331    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d8a943-a45d-434a-a400-eec464164ea2-istiod-ca-cert" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:30.814630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.814321    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-credential-socket" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.815575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.815539    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73d8a943-a45d-434a-a400-eec464164ea2-istio-podinfo" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:30.816226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.816204    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-kube-api-access-cr2fv" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "kube-api-access-cr2fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:30.816531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.816506    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-envoy" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:30.816907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.816889    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-istio-token" pod "73d8a943-a45d-434a-a400-eec464164ea2" (UID: "73d8a943-a45d-434a-a400-eec464164ea2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:30.953201 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.953151    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73d8a943-a45d-434a-a400-eec464164ea2-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.953201 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.953193    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73d8a943-a45d-434a-a400-eec464164ea2-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.953201 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.953208    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.953611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.953222    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.953611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.953235    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.953611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.953246    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73d8a943-a45d-434a-a400-eec464164ea2-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:30.953611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:30.953260    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr2fv\" (UniqueName: \"kubernetes.io/projected/73d8a943-a45d-434a-a400-eec464164ea2-kube-api-access-cr2fv\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:31.283511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.283461    4282 generic.go:356] "Generic (PLEG): container finished" podID="73d8a943-a45d-434a-a400-eec464164ea2" containerID="09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119" exitCode=0
Oct 30 22:23:31.283691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.283524    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" event={"ID":"73d8a943-a45d-434a-a400-eec464164ea2","Type":"ContainerDied","Data":"09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119"}
Oct 30 22:23:31.283691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.283580    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9" event={"ID":"73d8a943-a45d-434a-a400-eec464164ea2","Type":"ContainerDied","Data":"d618a8cc9e071886fbafea9d635ffa4c15f71a37026beaf8282b1055f823e842"}
Oct 30 22:23:31.283691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.283602    4282 scope.go:122] "RemoveContainer" containerID="09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119"
Oct 30 22:23:31.283691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.283538    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"
Oct 30 22:23:31.301564 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.301547    4282 scope.go:122] "RemoveContainer" containerID="eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00"
Oct 30 22:23:31.311786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.311761    4282 scope.go:122] "RemoveContainer" containerID="09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119"
Oct 30 22:23:31.312096 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:23:31.312075    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119\": container with ID starting with 09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119 not found: ID does not exist" containerID="09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119"
Oct 30 22:23:31.312192 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.312110    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119"} err="failed to get container status \"09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119\": rpc error: code = NotFound desc = could not find container \"09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119\": container with ID starting with 09f87e3aac95645273bc4cd6dea76b7c7a94d8ec4844de14fff15dacfa078119 not found: ID does not exist"
Oct 30 22:23:31.312192 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.312135    4282 scope.go:122] "RemoveContainer" containerID="eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00"
Oct 30 22:23:31.312435 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:23:31.312408    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00\": container with ID starting with eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00 not found: ID does not exist" containerID="eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00"
Oct 30 22:23:31.312517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.312443    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00"} err="failed to get container status \"eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00\": rpc error: code = NotFound desc = could not find container \"eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00\": container with ID starting with eda7d23bb087658034100cb47d536360f14abfea7316beef2c7a49cc90f95b00 not found: ID does not exist"
Oct 30 22:23:31.315293 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.315269    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"]
Oct 30 22:23:31.339487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.337591    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-v8mf9"]
Oct 30 22:23:31.466391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:31.466355    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d8a943-a45d-434a-a400-eec464164ea2" path="/var/lib/kubelet/pods/73d8a943-a45d-434a-a400-eec464164ea2/volumes"
Oct 30 22:23:34.651964 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:23:34.651925    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:23:44.641113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641083    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"]
Oct 30 22:23:44.641918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641903    4282 cpu_manager.go:414] "containerMap: removing container" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy"
Oct 30 22:23:44.641971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641918    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy"
Oct 30 22:23:44.641971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641933    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy"
Oct 30 22:23:44.641971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641938    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy"
Oct 30 22:23:44.641971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641944    4282 cpu_manager.go:414] "containerMap: removing container" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy"
Oct 30 22:23:44.641971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641949    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy"
Oct 30 22:23:44.641971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641958    4282 cpu_manager.go:414] "containerMap: removing container" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy"
Oct 30 22:23:44.641971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.641962    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642005    4282 cpu_manager.go:414] "containerMap: removing container" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642009    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642120    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642128    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="aded6eba-1991-4d9d-8e01-5cc2d7a0c74e" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642136    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642143    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642149    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642155    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642160    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e348f2cc-4f24-47ad-b0c6-41d667b29f78" containerName="istio-proxy"
Oct 30 22:23:44.642194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.642179    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy"
Oct 30 22:23:44.646152 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.646136    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:44.652509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.651652    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-c10cdb\"/\"default-dockercfg-jxtz2\""
Oct 30 22:23:44.652509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.651684    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-c10cdb\"/\"kube-root-ca.crt\""
Oct 30 22:23:44.652509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.651734    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-c10cdb\"/\"e2e-usage-sim-b15fa0-cert\""
Oct 30 22:23:44.652509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.651738    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-c10cdb\"/\"openshift-service-ca.crt\""
Oct 30 22:23:44.657525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.657498    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"]
Oct 30 22:23:44.728257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.728226    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-tls\") pod \"e2e-usage-sim-b15fa0-65f8d86c5c-965ks\" (UID: \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\") " pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:44.728504 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.728273    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn2sr\" (UniqueName: \"kubernetes.io/projected/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-kube-api-access-nn2sr\") pod \"e2e-usage-sim-b15fa0-65f8d86c5c-965ks\" (UID: \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\") " pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:44.830041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.830003    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-tls\") pod \"e2e-usage-sim-b15fa0-65f8d86c5c-965ks\" (UID: \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\") " pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:44.830254 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.830055    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn2sr\" (UniqueName: \"kubernetes.io/projected/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-kube-api-access-nn2sr\") pod \"e2e-usage-sim-b15fa0-65f8d86c5c-965ks\" (UID: \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\") " pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:44.832776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.832751    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-tls\") pod \"e2e-usage-sim-b15fa0-65f8d86c5c-965ks\" (UID: \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\") " pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:44.843043 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.843023    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn2sr\" (UniqueName: \"kubernetes.io/projected/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-kube-api-access-nn2sr\") pod \"e2e-usage-sim-b15fa0-65f8d86c5c-965ks\" (UID: \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\") " pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:44.962058 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:44.962029    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:45.291868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:45.291853    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"]
Oct 30 22:23:45.523553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:45.523512    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks" event={"ID":"8ad28078-8c55-4d57-a2cf-01aa6339c0cd","Type":"ContainerStarted","Data":"65bd7ab43ff176f7968998346e1b6e546ed9a514447111f245e96cbd09b8a309"}
Oct 30 22:23:45.523754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:45.523558    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks" event={"ID":"8ad28078-8c55-4d57-a2cf-01aa6339c0cd","Type":"ContainerStarted","Data":"36473e13bd7d0af9fb6df562233132e289d3436c26be291676dc8a81e347c8f3"}
Oct 30 22:23:45.523754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:45.523718    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:45.528313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:45.528291    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:23:45.540529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:45.540393    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks" podStartSLOduration=1.5403791770000002 podStartE2EDuration="1.540379177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:45.539198623 +0000 UTC m=+2644.197294867" watchObservedRunningTime="2026-10-30 22:23:45.540379177 +0000 UTC m=+2644.198475401"
Oct 30 22:23:46.723237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:46.723200    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"]
Oct 30 22:23:46.723869 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:46.723411    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy" containerID="cri-o://9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530" gracePeriod=30
Oct 30 22:23:52.264966 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.264919    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:23:52.302863 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.302824    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-socket\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.302943    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-data\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.302985    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-credential-socket\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303101    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istiod-ca-cert\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303383 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303354    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-envoy\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303405    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-podinfo\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303417    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istiod-ca-cert" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:52.303550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303448    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-certs\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303557    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-kube-api-access-gnkj2\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-kube-api-access-gnkj2\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303662    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-token\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-token\") pod \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\" (UID: \"3a92ac34-b27e-4d6d-8ed2-27f100247b6f\") "
Oct 30 22:23:52.303757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303731    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-data" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:52.303941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303783    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-socket" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:52.304015 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.303992    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-credential-socket" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:52.304776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.304751    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.304874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.304781    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.304874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.304796    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.304874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.304809    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.305047 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.304944    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-certs" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:52.306942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.306910    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-token" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:52.315999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.315958    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-podinfo" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:23:52.316265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.316241    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-kube-api-access-gnkj2" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "kube-api-access-gnkj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:52.316700 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.316671    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-envoy" pod "3a92ac34-b27e-4d6d-8ed2-27f100247b6f" (UID: "3a92ac34-b27e-4d6d-8ed2-27f100247b6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:23:52.407244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.407168    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.407244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.407208    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.407244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.407225    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.407244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.407239    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnkj2\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-kube-api-access-gnkj2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.407541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.407254    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a92ac34-b27e-4d6d-8ed2-27f100247b6f-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:52.765636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.765598    4282 generic.go:356] "Generic (PLEG): container finished" podID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerID="9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530" exitCode=0
Oct 30 22:23:52.765988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.765965    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" event={"ID":"3a92ac34-b27e-4d6d-8ed2-27f100247b6f","Type":"ContainerDied","Data":"9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530"}
Oct 30 22:23:52.766152 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.766136    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg" event={"ID":"3a92ac34-b27e-4d6d-8ed2-27f100247b6f","Type":"ContainerDied","Data":"5184fe54ca4f8073a55d5327f67d899c57e70f358fe9bbcea6a05f54eab75142"}
Oct 30 22:23:52.766303 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.766287    4282 scope.go:122] "RemoveContainer" containerID="9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530"
Oct 30 22:23:52.766626 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.766605    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"
Oct 30 22:23:52.796306 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.795960    4282 scope.go:122] "RemoveContainer" containerID="9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530"
Oct 30 22:23:52.796647 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:23:52.796617    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530\": container with ID starting with 9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530 not found: ID does not exist" containerID="9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530"
Oct 30 22:23:52.796769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.796655    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530"} err="failed to get container status \"9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530\": rpc error: code = NotFound desc = could not find container \"9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530\": container with ID starting with 9ea6f1338689b7afc78e1a4434649aa0ccb2e574d99bb1d65b12c2ce2f463530 not found: ID does not exist"
Oct 30 22:23:52.812374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.812349    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"]
Oct 30 22:23:52.820187 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:52.820163    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-4dmgg"]
Oct 30 22:23:53.466010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:53.465978    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" path="/var/lib/kubelet/pods/3a92ac34-b27e-4d6d-8ed2-27f100247b6f/volumes"
Oct 30 22:23:56.930581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:23:56.930546    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:24:04.689621 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:24:04.689568    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:24:34.879034 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:24:34.875648    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:24:51.190163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:24:51.190130    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:25:04.678500 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:25:04.676839    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:25:18.306492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:18.305885    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"]
Oct 30 22:25:18.345286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:18.344938    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"]
Oct 30 22:25:18.578250 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:18.578161    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"]
Oct 30 22:25:18.578457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:18.578428    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m" podUID="ed70ab40-99c6-412c-bf20-65031498ef71" containerName="maas-api" containerID="cri-o://879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5" gracePeriod=30
Oct 30 22:25:19.165960 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.165934    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:25:19.192748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.192724    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-maas-api-tls\") pod \"ed70ab40-99c6-412c-bf20-65031498ef71\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") "
Oct 30 22:25:19.192939 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.192807    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed70ab40-99c6-412c-bf20-65031498ef71-kube-api-access-22cxn\" (UniqueName: \"kubernetes.io/projected/ed70ab40-99c6-412c-bf20-65031498ef71-kube-api-access-22cxn\") pod \"ed70ab40-99c6-412c-bf20-65031498ef71\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") "
Oct 30 22:25:19.192939 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.192854    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-metrics-certs\") pod \"ed70ab40-99c6-412c-bf20-65031498ef71\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") "
Oct 30 22:25:19.192939 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.192921    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed70ab40-99c6-412c-bf20-65031498ef71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed70ab40-99c6-412c-bf20-65031498ef71-openshift-service-ca\") pod \"ed70ab40-99c6-412c-bf20-65031498ef71\" (UID: \"ed70ab40-99c6-412c-bf20-65031498ef71\") "
Oct 30 22:25:19.193417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.193378    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed70ab40-99c6-412c-bf20-65031498ef71-openshift-service-ca" pod "ed70ab40-99c6-412c-bf20-65031498ef71" (UID: "ed70ab40-99c6-412c-bf20-65031498ef71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:19.193714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.193691    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed70ab40-99c6-412c-bf20-65031498ef71-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:19.196816 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.196789    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-maas-api-tls" pod "ed70ab40-99c6-412c-bf20-65031498ef71" (UID: "ed70ab40-99c6-412c-bf20-65031498ef71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:19.197449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.197426    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed70ab40-99c6-412c-bf20-65031498ef71-kube-api-access-22cxn" pod "ed70ab40-99c6-412c-bf20-65031498ef71" (UID: "ed70ab40-99c6-412c-bf20-65031498ef71"). InnerVolumeSpecName "kube-api-access-22cxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:19.198314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.198287    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-metrics-certs" pod "ed70ab40-99c6-412c-bf20-65031498ef71" (UID: "ed70ab40-99c6-412c-bf20-65031498ef71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:19.305887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.305647    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:19.305887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.305690    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22cxn\" (UniqueName: \"kubernetes.io/projected/ed70ab40-99c6-412c-bf20-65031498ef71-kube-api-access-22cxn\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:19.305887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.305707    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed70ab40-99c6-412c-bf20-65031498ef71-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:19.417774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.417747    4282 generic.go:356] "Generic (PLEG): container finished" podID="ed70ab40-99c6-412c-bf20-65031498ef71" containerID="879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5" exitCode=0
Oct 30 22:25:19.418193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.417792    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"
Oct 30 22:25:19.418193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.417834    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m" event={"ID":"ed70ab40-99c6-412c-bf20-65031498ef71","Type":"ContainerDied","Data":"879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5"}
Oct 30 22:25:19.418193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.417893    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m" event={"ID":"ed70ab40-99c6-412c-bf20-65031498ef71","Type":"ContainerDied","Data":"607696ba113cbbaedd775c09677330a67c52ad7defa4799724badd33d3a5dc0d"}
Oct 30 22:25:19.418193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.417920    4282 scope.go:122] "RemoveContainer" containerID="879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5"
Oct 30 22:25:19.429486 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.429452    4282 scope.go:122] "RemoveContainer" containerID="879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5"
Oct 30 22:25:19.429734 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:25:19.429718    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5\": container with ID starting with 879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5 not found: ID does not exist" containerID="879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5"
Oct 30 22:25:19.429773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.429742    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5"} err="failed to get container status \"879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5\": rpc error: code = NotFound desc = could not find container \"879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5\": container with ID starting with 879ad82874b6970842f07e74dd52c4616a9880aa0d9e87e12bece8fe8773d5e5 not found: ID does not exist"
Oct 30 22:25:19.444992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.444963    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"]
Oct 30 22:25:19.449239 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.449216    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6f8222-85b96996d5-jlv2m"]
Oct 30 22:25:19.466685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:19.466661    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed70ab40-99c6-412c-bf20-65031498ef71" path="/var/lib/kubelet/pods/ed70ab40-99c6-412c-bf20-65031498ef71/volumes"
Oct 30 22:25:23.307140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:23.307040    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b" podUID="1913325b-12d7-4350-bb11-3207d9f870a6" containerName="payload-processing" containerID="cri-o://f11b8894c2685cbadadf803a4681389340c6051ecf122719c959ff5d74c975a8" gracePeriod=25
Oct 30 22:25:23.345733 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:23.345693    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f" podUID="ab57aefb-f4f7-4061-94f7-8effed875817" containerName="payload-pre-processing" containerID="cri-o://b59f4a0648fd110d994375f9652bff75293594e6aa6fae23925ebf86e350e0ae" gracePeriod=25
Oct 30 22:25:24.075695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:24.075660    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f" podUID="ab57aefb-f4f7-4061-94f7-8effed875817" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:33.046636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:33.046596    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b" podUID="1913325b-12d7-4350-bb11-3207d9f870a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:34.074447 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:34.074411    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f" podUID="ab57aefb-f4f7-4061-94f7-8effed875817" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:34.651010 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:25:34.650977    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:25:38.837706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.837669    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"]
Oct 30 22:25:38.838703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838679    4282 cpu_manager.go:414] "containerMap: removing container" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy"
Oct 30 22:25:38.838703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838703    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy"
Oct 30 22:25:38.838906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838744    4282 cpu_manager.go:414] "containerMap: removing container" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy"
Oct 30 22:25:38.838906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838752    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d8a943-a45d-434a-a400-eec464164ea2" containerName="istio-proxy"
Oct 30 22:25:38.838906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838767    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy"
Oct 30 22:25:38.838906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838773    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a025f32-89c7-4009-8b77-7ee839ed8c20" containerName="istio-proxy"
Oct 30 22:25:38.838906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838790    4282 cpu_manager.go:414] "containerMap: removing container" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy"
Oct 30 22:25:38.838906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838797    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579717b1-feef-4e2a-9da8-62721859adeb" containerName="istio-proxy"
Oct 30 22:25:38.838906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838830    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ed70ab40-99c6-412c-bf20-65031498ef71" containerName="maas-api"
Oct 30 22:25:38.838906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.838837    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed70ab40-99c6-412c-bf20-65031498ef71" containerName="maas-api"
Oct 30 22:25:38.839295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.839048    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed70ab40-99c6-412c-bf20-65031498ef71" containerName="maas-api"
Oct 30 22:25:38.839295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.839069    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a92ac34-b27e-4d6d-8ed2-27f100247b6f" containerName="istio-proxy"
Oct 30 22:25:38.844363 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.844343    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:38.853028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.852990    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4b2bb0\""
Oct 30 22:25:38.858812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.858786    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"]
Oct 30 22:25:38.863220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.863198    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"]
Oct 30 22:25:38.872620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.872594    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:38.880187 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.880166    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4b2bb0-dockercfg-hx557\""
Oct 30 22:25:38.890454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:38.890413    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"]
Oct 30 22:25:39.010517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.006065    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be12dee-de32-4f71-baad-46d448718612-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs\" (UID: \"1be12dee-de32-4f71-baad-46d448718612\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:39.010517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.006175    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6r5\" (UniqueName: \"kubernetes.io/projected/1be12dee-de32-4f71-baad-46d448718612-kube-api-access-ts6r5\") pod \"payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs\" (UID: \"1be12dee-de32-4f71-baad-46d448718612\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:39.010517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.006204    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bd8d\" (UniqueName: \"kubernetes.io/projected/a1a3ae17-f62e-4b49-83a6-6216fd549636-kube-api-access-5bd8d\") pod \"payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs\" (UID: \"a1a3ae17-f62e-4b49-83a6-6216fd549636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:39.010517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.006366    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a3ae17-f62e-4b49-83a6-6216fd549636-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs\" (UID: \"a1a3ae17-f62e-4b49-83a6-6216fd549636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:39.108666 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.108568    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a3ae17-f62e-4b49-83a6-6216fd549636-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs\" (UID: \"a1a3ae17-f62e-4b49-83a6-6216fd549636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:39.109129 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.109106    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be12dee-de32-4f71-baad-46d448718612-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs\" (UID: \"1be12dee-de32-4f71-baad-46d448718612\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:39.109311 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.109294    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6r5\" (UniqueName: \"kubernetes.io/projected/1be12dee-de32-4f71-baad-46d448718612-kube-api-access-ts6r5\") pod \"payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs\" (UID: \"1be12dee-de32-4f71-baad-46d448718612\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:39.112491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.109418    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bd8d\" (UniqueName: \"kubernetes.io/projected/a1a3ae17-f62e-4b49-83a6-6216fd549636-kube-api-access-5bd8d\") pod \"payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs\" (UID: \"a1a3ae17-f62e-4b49-83a6-6216fd549636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:39.142500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.114336    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a3ae17-f62e-4b49-83a6-6216fd549636-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs\" (UID: \"a1a3ae17-f62e-4b49-83a6-6216fd549636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:39.171493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.118352    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be12dee-de32-4f71-baad-46d448718612-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs\" (UID: \"1be12dee-de32-4f71-baad-46d448718612\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:39.171493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.152129    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6r5\" (UniqueName: \"kubernetes.io/projected/1be12dee-de32-4f71-baad-46d448718612-kube-api-access-ts6r5\") pod \"payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs\" (UID: \"1be12dee-de32-4f71-baad-46d448718612\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:39.171493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.158442    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bd8d\" (UniqueName: \"kubernetes.io/projected/a1a3ae17-f62e-4b49-83a6-6216fd549636-kube-api-access-5bd8d\") pod \"payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs\" (UID: \"a1a3ae17-f62e-4b49-83a6-6216fd549636\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:39.171493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.166736    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:39.192495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.191817    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:39.635081 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.635042    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"]
Oct 30 22:25:39.635989 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:25:39.635958    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be12dee_de32_4f71_baad_46d448718612.slice/crio-07270b374f45d03900f137bdbe1aab0711149074d4150a9fb88221a17b749f20 WatchSource:0}: Error finding container 07270b374f45d03900f137bdbe1aab0711149074d4150a9fb88221a17b749f20: Status 404 returned error can't find the container with id 07270b374f45d03900f137bdbe1aab0711149074d4150a9fb88221a17b749f20
Oct 30 22:25:39.638710 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:25:39.637861    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a3ae17_f62e_4b49_83a6_6216fd549636.slice/crio-c5cf76ced6d302f48448560cf17ab911cce823ee4c73dca5ad660e21aa6caa41 WatchSource:0}: Error finding container c5cf76ced6d302f48448560cf17ab911cce823ee4c73dca5ad660e21aa6caa41: Status 404 returned error can't find the container with id c5cf76ced6d302f48448560cf17ab911cce823ee4c73dca5ad660e21aa6caa41
Oct 30 22:25:39.639950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.639931    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"]
Oct 30 22:25:39.651880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.651819    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs" event={"ID":"1be12dee-de32-4f71-baad-46d448718612","Type":"ContainerStarted","Data":"07270b374f45d03900f137bdbe1aab0711149074d4150a9fb88221a17b749f20"}
Oct 30 22:25:39.653200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.653170    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs" event={"ID":"a1a3ae17-f62e-4b49-83a6-6216fd549636","Type":"ContainerStarted","Data":"c5cf76ced6d302f48448560cf17ab911cce823ee4c73dca5ad660e21aa6caa41"}
Oct 30 22:25:39.907960 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.907576    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"]
Oct 30 22:25:39.921007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.920976    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:39.941887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.941854    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"]
Oct 30 22:25:39.944306 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.944280    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c92aec\""
Oct 30 22:25:39.947335 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.947309    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nggdf\" (UniqueName: \"kubernetes.io/projected/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-kube-api-access-nggdf\") pod \"payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592\" (UID: \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:39.947542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.947523    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592\" (UID: \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:39.974862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.974835    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"]
Oct 30 22:25:39.979647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.979627    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:39.986743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:39.986722    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c92aec-dockercfg-5xjmx\""
Oct 30 22:25:40.003987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.003955    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"]
Oct 30 22:25:40.072083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.056071    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592\" (UID: \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:40.090534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.073066    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592\" (UID: \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:40.119507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.056180    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x695n\" (UniqueName: \"kubernetes.io/projected/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-kube-api-access-x695n\") pod \"payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx\" (UID: \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:40.119507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.118090    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx\" (UID: \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:40.119507 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.118167    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nggdf\" (UniqueName: \"kubernetes.io/projected/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-kube-api-access-nggdf\") pod \"payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592\" (UID: \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:40.244490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.241293    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nggdf\" (UniqueName: \"kubernetes.io/projected/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-kube-api-access-nggdf\") pod \"payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592\" (UID: \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:40.264491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.258608    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x695n\" (UniqueName: \"kubernetes.io/projected/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-kube-api-access-x695n\") pod \"payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx\" (UID: \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:40.264491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.258901    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx\" (UID: \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:40.275490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.265251    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx\" (UID: \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:40.303499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.300189    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x695n\" (UniqueName: \"kubernetes.io/projected/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-kube-api-access-x695n\") pod \"payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx\" (UID: \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:40.541409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.539829    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:40.595172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.595145    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:40.670692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.670590    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs" event={"ID":"a1a3ae17-f62e-4b49-83a6-6216fd549636","Type":"ContainerStarted","Data":"164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e"}
Oct 30 22:25:40.670692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.670660    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:40.675856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.675122    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs" event={"ID":"1be12dee-de32-4f71-baad-46d448718612","Type":"ContainerStarted","Data":"680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe"}
Oct 30 22:25:40.675856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.675710    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:40.702966 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.697368    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs" podStartSLOduration=2.697351604 podStartE2EDuration="2.697351604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:40.69600186 +0000 UTC m=+2759.354098084" watchObservedRunningTime="2026-10-30 22:25:40.697351604 +0000 UTC m=+2759.355447828"
Oct 30 22:25:40.727437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.727221    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs" podStartSLOduration=2.727205684 podStartE2EDuration="2.727205684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:40.718090836 +0000 UTC m=+2759.376187059" watchObservedRunningTime="2026-10-30 22:25:40.727205684 +0000 UTC m=+2759.385301908"
Oct 30 22:25:40.883974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.876808    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"]
Oct 30 22:25:40.922219 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:40.922191    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"]
Oct 30 22:25:40.927850 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:25:40.927824    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303a8976_dc91_43cf_ae4e_4e4cfcd2f8aa.slice/crio-1bef3bfc79054080e46e9706912ac275af61eba0ab1364bafc5afdf661c01404 WatchSource:0}: Error finding container 1bef3bfc79054080e46e9706912ac275af61eba0ab1364bafc5afdf661c01404: Status 404 returned error can't find the container with id 1bef3bfc79054080e46e9706912ac275af61eba0ab1364bafc5afdf661c01404
Oct 30 22:25:41.684345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:41.684311    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx" event={"ID":"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa","Type":"ContainerStarted","Data":"a0356b309a9659226cacda5d202852280ce27b66e719d5cc060f657e3253cd8c"}
Oct 30 22:25:41.684610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:41.684378    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx" event={"ID":"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa","Type":"ContainerStarted","Data":"1bef3bfc79054080e46e9706912ac275af61eba0ab1364bafc5afdf661c01404"}
Oct 30 22:25:41.684975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:41.684952    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:41.686268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:41.686240    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592" event={"ID":"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e","Type":"ContainerStarted","Data":"48933d7f3cad28e1acf8fea2aeeaa0314669484c87bb6ef224162a510135fc0e"}
Oct 30 22:25:41.686374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:41.686289    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592" event={"ID":"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e","Type":"ContainerStarted","Data":"21c9703389b0fc5b7764d19ae8d2cf0c716a0645bbea4ab9bcba0a80dc8b85ab"}
Oct 30 22:25:41.686374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:41.686320    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:41.749689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:41.749648    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx" podStartSLOduration=2.749635643 podStartE2EDuration="2.749635643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:41.748816759 +0000 UTC m=+2760.406912981" watchObservedRunningTime="2026-10-30 22:25:41.749635643 +0000 UTC m=+2760.407731865"
Oct 30 22:25:41.780547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:41.780495    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592" podStartSLOduration=2.780461969 podStartE2EDuration="2.780461969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:41.778485296 +0000 UTC m=+2760.436581515" watchObservedRunningTime="2026-10-30 22:25:41.780461969 +0000 UTC m=+2760.438558192"
Oct 30 22:25:43.046212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.046176    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b" podUID="1913325b-12d7-4350-bb11-3207d9f870a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:25:43.743236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.743200    4282 generic.go:356] "Generic (PLEG): container finished" podID="ab57aefb-f4f7-4061-94f7-8effed875817" containerID="b59f4a0648fd110d994375f9652bff75293594e6aa6fae23925ebf86e350e0ae" exitCode=0
Oct 30 22:25:43.743384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.743274    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f" event={"ID":"ab57aefb-f4f7-4061-94f7-8effed875817","Type":"ContainerDied","Data":"b59f4a0648fd110d994375f9652bff75293594e6aa6fae23925ebf86e350e0ae"}
Oct 30 22:25:43.744658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.744636    4282 generic.go:356] "Generic (PLEG): container finished" podID="1913325b-12d7-4350-bb11-3207d9f870a6" containerID="f11b8894c2685cbadadf803a4681389340c6051ecf122719c959ff5d74c975a8" exitCode=0
Oct 30 22:25:43.745533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.745495    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b" event={"ID":"1913325b-12d7-4350-bb11-3207d9f870a6","Type":"ContainerDied","Data":"f11b8894c2685cbadadf803a4681389340c6051ecf122719c959ff5d74c975a8"}
Oct 30 22:25:43.765479 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.765446    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:25:43.769073 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.769054    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:25:43.821624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.821593    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1913325b-12d7-4350-bb11-3207d9f870a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1913325b-12d7-4350-bb11-3207d9f870a6-plugins-config-volume\") pod \"1913325b-12d7-4350-bb11-3207d9f870a6\" (UID: \"1913325b-12d7-4350-bb11-3207d9f870a6\") "
Oct 30 22:25:43.821787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.821653    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1913325b-12d7-4350-bb11-3207d9f870a6-kube-api-access-zjdws\" (UniqueName: \"kubernetes.io/projected/1913325b-12d7-4350-bb11-3207d9f870a6-kube-api-access-zjdws\") pod \"1913325b-12d7-4350-bb11-3207d9f870a6\" (UID: \"1913325b-12d7-4350-bb11-3207d9f870a6\") "
Oct 30 22:25:43.821787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.821684    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab57aefb-f4f7-4061-94f7-8effed875817-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab57aefb-f4f7-4061-94f7-8effed875817-plugins-config-volume\") pod \"ab57aefb-f4f7-4061-94f7-8effed875817\" (UID: \"ab57aefb-f4f7-4061-94f7-8effed875817\") "
Oct 30 22:25:43.821874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.821809    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab57aefb-f4f7-4061-94f7-8effed875817-kube-api-access-qxt7c\" (UniqueName: \"kubernetes.io/projected/ab57aefb-f4f7-4061-94f7-8effed875817-kube-api-access-qxt7c\") pod \"ab57aefb-f4f7-4061-94f7-8effed875817\" (UID: \"ab57aefb-f4f7-4061-94f7-8effed875817\") "
Oct 30 22:25:43.822097 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.821969    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1913325b-12d7-4350-bb11-3207d9f870a6-plugins-config-volume" pod "1913325b-12d7-4350-bb11-3207d9f870a6" (UID: "1913325b-12d7-4350-bb11-3207d9f870a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:43.822199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.822160    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab57aefb-f4f7-4061-94f7-8effed875817-plugins-config-volume" pod "ab57aefb-f4f7-4061-94f7-8effed875817" (UID: "ab57aefb-f4f7-4061-94f7-8effed875817"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:43.822671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.822648    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab57aefb-f4f7-4061-94f7-8effed875817-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:43.822671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.822672    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1913325b-12d7-4350-bb11-3207d9f870a6-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:43.824788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.824770    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1913325b-12d7-4350-bb11-3207d9f870a6-kube-api-access-zjdws" pod "1913325b-12d7-4350-bb11-3207d9f870a6" (UID: "1913325b-12d7-4350-bb11-3207d9f870a6"). InnerVolumeSpecName "kube-api-access-zjdws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:43.824882 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.824795    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab57aefb-f4f7-4061-94f7-8effed875817-kube-api-access-qxt7c" pod "ab57aefb-f4f7-4061-94f7-8effed875817" (UID: "ab57aefb-f4f7-4061-94f7-8effed875817"). InnerVolumeSpecName "kube-api-access-qxt7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:43.924114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.924077    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjdws\" (UniqueName: \"kubernetes.io/projected/1913325b-12d7-4350-bb11-3207d9f870a6-kube-api-access-zjdws\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:43.924114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:43.924110    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxt7c\" (UniqueName: \"kubernetes.io/projected/ab57aefb-f4f7-4061-94f7-8effed875817-kube-api-access-qxt7c\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:44.754078 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.754048    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"
Oct 30 22:25:44.754554 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.754046    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f" event={"ID":"ab57aefb-f4f7-4061-94f7-8effed875817","Type":"ContainerDied","Data":"3c2f0a0e10a4aecd591eed5ca2f62bda2e5e27689f7db103cda57a671b33468c"}
Oct 30 22:25:44.754554 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.754182    4282 scope.go:122] "RemoveContainer" containerID="b59f4a0648fd110d994375f9652bff75293594e6aa6fae23925ebf86e350e0ae"
Oct 30 22:25:44.755685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.755450    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"
Oct 30 22:25:44.755685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.755462    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b" event={"ID":"1913325b-12d7-4350-bb11-3207d9f870a6","Type":"ContainerDied","Data":"71cb535f88fff0f7654dd4d01cbfd0a2e71cf297c57c8dad4ecabbf656b20e4f"}
Oct 30 22:25:44.765778 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.765761    4282 scope.go:122] "RemoveContainer" containerID="f11b8894c2685cbadadf803a4681389340c6051ecf122719c959ff5d74c975a8"
Oct 30 22:25:44.789256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.789223    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"]
Oct 30 22:25:44.793612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.793590    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6f8222-5dc449fb67-w9k6f"]
Oct 30 22:25:44.802517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.802494    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"]
Oct 30 22:25:44.808718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:44.808697    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6f8222-868585bdcf-bwp8b"]
Oct 30 22:25:45.465656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:45.465626    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1913325b-12d7-4350-bb11-3207d9f870a6" path="/var/lib/kubelet/pods/1913325b-12d7-4350-bb11-3207d9f870a6/volumes"
Oct 30 22:25:45.466133 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:45.466110    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab57aefb-f4f7-4061-94f7-8effed875817" path="/var/lib/kubelet/pods/ab57aefb-f4f7-4061-94f7-8effed875817/volumes"
Oct 30 22:25:51.690191 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:51.690164    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:25:52.698723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:52.698688    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:25:52.699249 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:52.698770    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:25:53.521051 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:53.521009    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:25:53.787549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:53.782651    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:25:58.408479 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:58.408425    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"]
Oct 30 22:25:58.420672 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:25:58.420637    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"]
Oct 30 22:26:00.508149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.508117    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"]
Oct 30 22:26:00.508924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.508907    4282 cpu_manager.go:414] "containerMap: removing container" podUID="1913325b-12d7-4350-bb11-3207d9f870a6" containerName="payload-processing"
Oct 30 22:26:00.508924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.508926    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1913325b-12d7-4350-bb11-3207d9f870a6" containerName="payload-processing"
Oct 30 22:26:00.509031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.508955    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ab57aefb-f4f7-4061-94f7-8effed875817" containerName="payload-pre-processing"
Oct 30 22:26:00.509031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.508961    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab57aefb-f4f7-4061-94f7-8effed875817" containerName="payload-pre-processing"
Oct 30 22:26:00.509106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.509078    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1913325b-12d7-4350-bb11-3207d9f870a6" containerName="payload-processing"
Oct 30 22:26:00.509106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.509093    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab57aefb-f4f7-4061-94f7-8effed875817" containerName="payload-pre-processing"
Oct 30 22:26:00.513445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.513428    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:00.520005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.519984    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-58d5fa\""
Oct 30 22:26:00.520150 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.520039    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-58d5fa-dockercfg-xkjdr\""
Oct 30 22:26:00.526151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.526131    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"]
Oct 30 22:26:00.531439 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.531403    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"]
Oct 30 22:26:00.536822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.536797    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:00.566484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.566434    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"]
Oct 30 22:26:00.729493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.725602    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d7feb5-8074-4993-ae62-5352da049922-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh\" (UID: \"31d7feb5-8074-4993-ae62-5352da049922\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:00.729493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.725702    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cvj\" (UniqueName: \"kubernetes.io/projected/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-kube-api-access-n4cvj\") pod \"payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8\" (UID: \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:00.729493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.725808    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8\" (UID: \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:00.729493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.725863    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlfv\" (UniqueName: \"kubernetes.io/projected/31d7feb5-8074-4993-ae62-5352da049922-kube-api-access-hhlfv\") pod \"payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh\" (UID: \"31d7feb5-8074-4993-ae62-5352da049922\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:00.839651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.839555    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cvj\" (UniqueName: \"kubernetes.io/projected/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-kube-api-access-n4cvj\") pod \"payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8\" (UID: \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:00.840502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.840459    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8\" (UID: \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:00.841600 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.841579    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlfv\" (UniqueName: \"kubernetes.io/projected/31d7feb5-8074-4993-ae62-5352da049922-kube-api-access-hhlfv\") pod \"payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh\" (UID: \"31d7feb5-8074-4993-ae62-5352da049922\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:00.842354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.842330    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d7feb5-8074-4993-ae62-5352da049922-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh\" (UID: \"31d7feb5-8074-4993-ae62-5352da049922\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:00.843434 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.841502    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8\" (UID: \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:00.843609 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.843253    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d7feb5-8074-4993-ae62-5352da049922-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh\" (UID: \"31d7feb5-8074-4993-ae62-5352da049922\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:00.889647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.889563    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cvj\" (UniqueName: \"kubernetes.io/projected/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-kube-api-access-n4cvj\") pod \"payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8\" (UID: \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:00.941645 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:00.941606    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlfv\" (UniqueName: \"kubernetes.io/projected/31d7feb5-8074-4993-ae62-5352da049922-kube-api-access-hhlfv\") pod \"payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh\" (UID: \"31d7feb5-8074-4993-ae62-5352da049922\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:01.132238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:01.132146    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:01.194365 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:01.193045    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:01.447111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:01.447084    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"]
Oct 30 22:26:01.447814 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:26:01.447787    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d7feb5_8074_4993_ae62_5352da049922.slice/crio-0199064e6dac92c900ed40b426a00dda8fb8e715b8de7d02a8b79f83c6730963 WatchSource:0}: Error finding container 0199064e6dac92c900ed40b426a00dda8fb8e715b8de7d02a8b79f83c6730963: Status 404 returned error can't find the container with id 0199064e6dac92c900ed40b426a00dda8fb8e715b8de7d02a8b79f83c6730963
Oct 30 22:26:01.549539 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:26:01.549512    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda715bc3b_bebf_4a86_b0dc_ffd695a2854b.slice/crio-8e15e61e5c2462b0ee342f84d1ff01fd081eabd18bcd2fe7be07b8f5e70336f1 WatchSource:0}: Error finding container 8e15e61e5c2462b0ee342f84d1ff01fd081eabd18bcd2fe7be07b8f5e70336f1: Status 404 returned error can't find the container with id 8e15e61e5c2462b0ee342f84d1ff01fd081eabd18bcd2fe7be07b8f5e70336f1
Oct 30 22:26:01.550529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:01.550501    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"]
Oct 30 22:26:02.166740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:02.166702    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh" event={"ID":"31d7feb5-8074-4993-ae62-5352da049922","Type":"ContainerStarted","Data":"015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1"}
Oct 30 22:26:02.166949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:02.166762    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh" event={"ID":"31d7feb5-8074-4993-ae62-5352da049922","Type":"ContainerStarted","Data":"0199064e6dac92c900ed40b426a00dda8fb8e715b8de7d02a8b79f83c6730963"}
Oct 30 22:26:02.166949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:02.166794    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:02.168433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:02.168402    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8" event={"ID":"a715bc3b-bebf-4a86-b0dc-ffd695a2854b","Type":"ContainerStarted","Data":"666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06"}
Oct 30 22:26:02.168586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:02.168455    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8" event={"ID":"a715bc3b-bebf-4a86-b0dc-ffd695a2854b","Type":"ContainerStarted","Data":"8e15e61e5c2462b0ee342f84d1ff01fd081eabd18bcd2fe7be07b8f5e70336f1"}
Oct 30 22:26:02.168586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:02.168520    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:02.191091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:02.191047    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh" podStartSLOduration=2.191033961 podStartE2EDuration="2.191033961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:02.18981048 +0000 UTC m=+2780.847906702" watchObservedRunningTime="2026-10-30 22:26:02.191033961 +0000 UTC m=+2780.849130186"
Oct 30 22:26:02.211180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:02.211134    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8" podStartSLOduration=2.211121713 podStartE2EDuration="2.211121713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:02.207023926 +0000 UTC m=+2780.865120149" watchObservedRunningTime="2026-10-30 22:26:02.211121713 +0000 UTC m=+2780.869217935"
Oct 30 22:26:03.411492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.410529    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr" podUID="3c92a189-3606-4480-bb2d-d1a2281b4ea5" containerName="payload-processing" containerID="cri-o://440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1" gracePeriod=25
Oct 30 22:26:03.444130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.428433    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz" podUID="78322197-56a6-4b99-9e53-ab726a8fea53" containerName="payload-pre-processing" containerID="cri-o://e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61" gracePeriod=25
Oct 30 22:26:03.835258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.835231    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:26:03.855772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.855739    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:26:03.915117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.915078    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78322197-56a6-4b99-9e53-ab726a8fea53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78322197-56a6-4b99-9e53-ab726a8fea53-plugins-config-volume\") pod \"78322197-56a6-4b99-9e53-ab726a8fea53\" (UID: \"78322197-56a6-4b99-9e53-ab726a8fea53\") "
Oct 30 22:26:03.915302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.915285    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c92a189-3606-4480-bb2d-d1a2281b4ea5-kube-api-access-hj98w\" (UniqueName: \"kubernetes.io/projected/3c92a189-3606-4480-bb2d-d1a2281b4ea5-kube-api-access-hj98w\") pod \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\" (UID: \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\") "
Oct 30 22:26:03.915392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.915378    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c92a189-3606-4480-bb2d-d1a2281b4ea5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c92a189-3606-4480-bb2d-d1a2281b4ea5-plugins-config-volume\") pod \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\" (UID: \"3c92a189-3606-4480-bb2d-d1a2281b4ea5\") "
Oct 30 22:26:03.915536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.915519    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78322197-56a6-4b99-9e53-ab726a8fea53-kube-api-access-bkdvt\" (UniqueName: \"kubernetes.io/projected/78322197-56a6-4b99-9e53-ab726a8fea53-kube-api-access-bkdvt\") pod \"78322197-56a6-4b99-9e53-ab726a8fea53\" (UID: \"78322197-56a6-4b99-9e53-ab726a8fea53\") "
Oct 30 22:26:03.915621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.915548    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78322197-56a6-4b99-9e53-ab726a8fea53-plugins-config-volume" pod "78322197-56a6-4b99-9e53-ab726a8fea53" (UID: "78322197-56a6-4b99-9e53-ab726a8fea53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:03.916025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.915997    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c92a189-3606-4480-bb2d-d1a2281b4ea5-plugins-config-volume" pod "3c92a189-3606-4480-bb2d-d1a2281b4ea5" (UID: "3c92a189-3606-4480-bb2d-d1a2281b4ea5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:03.916317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.916290    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c92a189-3606-4480-bb2d-d1a2281b4ea5-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:03.916452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.916320    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78322197-56a6-4b99-9e53-ab726a8fea53-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:03.917971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.917942    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c92a189-3606-4480-bb2d-d1a2281b4ea5-kube-api-access-hj98w" pod "3c92a189-3606-4480-bb2d-d1a2281b4ea5" (UID: "3c92a189-3606-4480-bb2d-d1a2281b4ea5"). InnerVolumeSpecName "kube-api-access-hj98w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:03.918455 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:03.918437    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78322197-56a6-4b99-9e53-ab726a8fea53-kube-api-access-bkdvt" pod "78322197-56a6-4b99-9e53-ab726a8fea53" (UID: "78322197-56a6-4b99-9e53-ab726a8fea53"). InnerVolumeSpecName "kube-api-access-bkdvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:04.017456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.017383    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj98w\" (UniqueName: \"kubernetes.io/projected/3c92a189-3606-4480-bb2d-d1a2281b4ea5-kube-api-access-hj98w\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:04.017456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.017414    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdvt\" (UniqueName: \"kubernetes.io/projected/78322197-56a6-4b99-9e53-ab726a8fea53-kube-api-access-bkdvt\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:04.186112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.186074    4282 generic.go:356] "Generic (PLEG): container finished" podID="3c92a189-3606-4480-bb2d-d1a2281b4ea5" containerID="440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1" exitCode=0
Oct 30 22:26:04.186312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.186126    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"
Oct 30 22:26:04.186312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.186145    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr" event={"ID":"3c92a189-3606-4480-bb2d-d1a2281b4ea5","Type":"ContainerDied","Data":"440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1"}
Oct 30 22:26:04.186312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.186217    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr" event={"ID":"3c92a189-3606-4480-bb2d-d1a2281b4ea5","Type":"ContainerDied","Data":"4ff774cdb491c9b2ca1164946e23e0e131bd43c7e5f3591d4910fdbb735811ab"}
Oct 30 22:26:04.186312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.186245    4282 scope.go:122] "RemoveContainer" containerID="440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1"
Oct 30 22:26:04.187710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.187677    4282 generic.go:356] "Generic (PLEG): container finished" podID="78322197-56a6-4b99-9e53-ab726a8fea53" containerID="e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61" exitCode=0
Oct 30 22:26:04.187850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.187728    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"
Oct 30 22:26:04.187850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.187784    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz" event={"ID":"78322197-56a6-4b99-9e53-ab726a8fea53","Type":"ContainerDied","Data":"e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61"}
Oct 30 22:26:04.187962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.187827    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz" event={"ID":"78322197-56a6-4b99-9e53-ab726a8fea53","Type":"ContainerDied","Data":"1bc5e10e8e1f7d8c3047edd94494d5795c0bf9c123816f3f530b88e8d6280d3f"}
Oct 30 22:26:04.198137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.198115    4282 scope.go:122] "RemoveContainer" containerID="440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1"
Oct 30 22:26:04.198407 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:26:04.198388    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1\": container with ID starting with 440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1 not found: ID does not exist" containerID="440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1"
Oct 30 22:26:04.198500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.198415    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1"} err="failed to get container status \"440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1\": rpc error: code = NotFound desc = could not find container \"440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1\": container with ID starting with 440de34245de16570c5139217012f41bacc0b963a26825e8f0a1fa38d234b0b1 not found: ID does not exist"
Oct 30 22:26:04.198500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.198432    4282 scope.go:122] "RemoveContainer" containerID="e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61"
Oct 30 22:26:04.209060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.209044    4282 scope.go:122] "RemoveContainer" containerID="e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61"
Oct 30 22:26:04.209337 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:26:04.209315    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61\": container with ID starting with e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61 not found: ID does not exist" containerID="e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61"
Oct 30 22:26:04.209431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.209342    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61"} err="failed to get container status \"e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61\": rpc error: code = NotFound desc = could not find container \"e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61\": container with ID starting with e4208ec114a9539f01d118518e498a6be2af02d9c757eabb3f9c049fb7aebf61 not found: ID does not exist"
Oct 30 22:26:04.225047 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.225028    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"]
Oct 30 22:26:04.229261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.229241    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d8c835-5d7678b7dc-w7plr"]
Oct 30 22:26:04.240481 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.240451    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"]
Oct 30 22:26:04.253730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:04.253711    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d8c835-7795769895-wx6zz"]
Oct 30 22:26:04.650293 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:26:04.650260    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:26:05.465819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:05.465792    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c92a189-3606-4480-bb2d-d1a2281b4ea5" path="/var/lib/kubelet/pods/3c92a189-3606-4480-bb2d-d1a2281b4ea5/volumes"
Oct 30 22:26:05.468040 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:05.466212    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78322197-56a6-4b99-9e53-ab726a8fea53" path="/var/lib/kubelet/pods/78322197-56a6-4b99-9e53-ab726a8fea53/volumes"
Oct 30 22:26:12.801482 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.801441    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"]
Oct 30 22:26:12.802527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.802504    4282 cpu_manager.go:414] "containerMap: removing container" podUID="78322197-56a6-4b99-9e53-ab726a8fea53" containerName="payload-pre-processing"
Oct 30 22:26:12.802527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.802527    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78322197-56a6-4b99-9e53-ab726a8fea53" containerName="payload-pre-processing"
Oct 30 22:26:12.802727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.802557    4282 cpu_manager.go:414] "containerMap: removing container" podUID="3c92a189-3606-4480-bb2d-d1a2281b4ea5" containerName="payload-processing"
Oct 30 22:26:12.802727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.802564    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c92a189-3606-4480-bb2d-d1a2281b4ea5" containerName="payload-processing"
Oct 30 22:26:12.802844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.802746    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="78322197-56a6-4b99-9e53-ab726a8fea53" containerName="payload-pre-processing"
Oct 30 22:26:12.802844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.802766    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c92a189-3606-4480-bb2d-d1a2281b4ea5" containerName="payload-processing"
Oct 30 22:26:12.806927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.806907    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.817029 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.817009    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-120cb1-openshift-default-dockercfg-9p94d\""
Oct 30 22:26:12.822657 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.822636    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"]
Oct 30 22:26:12.861426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.861392    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.861630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.861443    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.861716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.861625    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hrdb\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-kube-api-access-6hrdb\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.861716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.861677    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.861905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.861865    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.861993 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.861905    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.862050 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.862009    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.862114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.862047    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.862219 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.862155    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963030 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.962998    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hrdb\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-kube-api-access-6hrdb\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963043    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963129    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963147    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963199    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963226    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963340 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963286    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963481    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963594 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963521    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963594 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963575    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963617    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963786    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.963879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963851    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.964112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.963988    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.965748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.965725    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.966189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.966166    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.977349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.977324    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hrdb\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-kube-api-access-6hrdb\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:12.977446 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:12.977414    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:13.123091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.123002    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:13.182086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.179666    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:26:13.182086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.180827    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:26:13.430349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.430323    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"]
Oct 30 22:26:13.430509 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:26:13.430450    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a61cf4_e93a_4fad_a160_e6f385fcf88f.slice/crio-44cd1fd0b4305201e163cd79e1f2b4d7159a1d67a2909bcc07553e9457d25e25 WatchSource:0}: Error finding container 44cd1fd0b4305201e163cd79e1f2b4d7159a1d67a2909bcc07553e9457d25e25: Status 404 returned error can't find the container with id 44cd1fd0b4305201e163cd79e1f2b4d7159a1d67a2909bcc07553e9457d25e25
Oct 30 22:26:13.433753 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.433720    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:13.433831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.433787    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:13.433831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.433816    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:13.888024 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.887936    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"]
Oct 30 22:26:13.893251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.893220    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:13.900625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.900603    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-df200a-openshift-default-dockercfg-rrfqx\""
Oct 30 22:26:13.965987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.963602    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"]
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:13.999997    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.000041    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.000106    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.000155    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.000289    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qnh\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-kube-api-access-v4qnh\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.000398    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.000439    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.000482    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.004766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.000508    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.101561 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101525    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.101757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101573    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.101757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101594    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.101757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101612    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.101757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101684    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.101757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101700    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.101757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101735    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.102092 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101771    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.102092 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.101883    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qnh\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-kube-api-access-v4qnh\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.102204 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.102128    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.102528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.102493    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.102878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.102556    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.102878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.102576    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.102878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.102699    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.104762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.104744    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.105168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.105150    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.113110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.113080    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qnh\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-kube-api-access-v4qnh\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.113998 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.113982    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.249497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.239917    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:14.509923 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.509382    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" event={"ID":"17a61cf4-e93a-4fad-a160-e6f385fcf88f","Type":"ContainerStarted","Data":"e84bbca860e5e2fea50d5be7cb5d99e5b1716d58886e3598478e63b1da19edc3"}
Oct 30 22:26:14.509923 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.509447    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" event={"ID":"17a61cf4-e93a-4fad-a160-e6f385fcf88f","Type":"ContainerStarted","Data":"44cd1fd0b4305201e163cd79e1f2b4d7159a1d67a2909bcc07553e9457d25e25"}
Oct 30 22:26:14.589738 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.589672    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" podStartSLOduration=2.589650207 podStartE2EDuration="2.589650207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:14.585304206 +0000 UTC m=+2793.243400451" watchObservedRunningTime="2026-10-30 22:26:14.589650207 +0000 UTC m=+2793.247746430"
Oct 30 22:26:14.786977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.786877    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"]
Oct 30 22:26:14.791758 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:26:14.791732    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963db1f0_ee51_4b7d_a0fc_c3913e5cce2e.slice/crio-427a437074f3b008f15deb90a2ea5aca3c21a261b97b00e32dbe01f2a9748cb0 WatchSource:0}: Error finding container 427a437074f3b008f15deb90a2ea5aca3c21a261b97b00e32dbe01f2a9748cb0: Status 404 returned error can't find the container with id 427a437074f3b008f15deb90a2ea5aca3c21a261b97b00e32dbe01f2a9748cb0
Oct 30 22:26:14.795387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.795352    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:14.795498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.795428    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:14.795498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:14.795459    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:15.124180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:15.124090    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:15.125737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:15.125707    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:26:15.125884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:15.125754    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:26:15.538591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:15.538556    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" event={"ID":"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e","Type":"ContainerStarted","Data":"c5383f9420afadfc42e2da6d068fe67c30988dfdbd90391ed4ec548c213bc6e3"}
Oct 30 22:26:15.538801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:15.538611    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" event={"ID":"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e","Type":"ContainerStarted","Data":"427a437074f3b008f15deb90a2ea5aca3c21a261b97b00e32dbe01f2a9748cb0"}
Oct 30 22:26:15.567493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:15.564405    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podStartSLOduration=2.564384307 podStartE2EDuration="2.564384307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:15.560160433 +0000 UTC m=+2794.218256659" watchObservedRunningTime="2026-10-30 22:26:15.564384307 +0000 UTC m=+2794.222480530"
Oct 30 22:26:16.128493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:16.125995    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:26:16.128493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:16.126072    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:26:16.241559 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:16.241441    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:16.243248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:16.243217    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:26:16.243319 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:16.243286    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 22:26:17.123919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:17.123885    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:26:17.124130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:17.123962    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:26:17.241165 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:17.241133    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:26:17.241624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:17.241181    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 22:26:18.123958 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:18.123921    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:26:18.124182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:18.123997    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:26:18.240888 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:18.240857    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:26:18.241062 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:18.240901    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 22:26:19.123491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:19.123440    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:26:19.124102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:19.124064    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:26:19.241814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:19.240977    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:26:19.241814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:19.241037    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 22:26:20.124577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:20.124545    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused" start-of-body=
Oct 30 22:26:20.124975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:20.124609    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.245:15021/healthz/ready\": dial tcp 10.133.0.245:15021: connect: connection refused"
Oct 30 22:26:20.242261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:20.242226    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:26:20.242445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:20.242281    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 22:26:21.129371 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:21.129331    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:21.129919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:21.129889    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:21.130685 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:21.130667    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:26:21.241326 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:21.241290    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 30 22:26:21.241958 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:21.241354    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 30 22:26:22.272381 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:22.272340    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:22.273009 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:22.272807    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:22.273587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:22.273567    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:26:29.563357 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.563317    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"]
Oct 30 22:26:29.572925 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.572900    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.578699 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.578672    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"]
Oct 30 22:26:29.580587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.580455    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-120cb1\"/\"default-dockercfg-5vgt7\""
Oct 30 22:26:29.580587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.580539    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-120cb1\"/\"openshift-service-ca.crt\""
Oct 30 22:26:29.580587 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.580565    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-120cb1\"/\"kube-root-ca.crt\""
Oct 30 22:26:29.580851 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.580777    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-120cb1\"/\"test-model-120cb1-kserve-self-signed-certs\""
Oct 30 22:26:29.583506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.583482    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3156f5-7370-486a-89fb-65087b1f969e-tls-certs\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.584099 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.584072    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-tmp-dir\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.584220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.584208    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-dshm\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.584341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.584323    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-home\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.584689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.584670    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-model-cache\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.584807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.584695    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2n69\" (UniqueName: \"kubernetes.io/projected/bc3156f5-7370-486a-89fb-65087b1f969e-kube-api-access-x2n69\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.686311 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.686266    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-home\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.686529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.686403    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-model-cache\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.686529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.686428    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2n69\" (UniqueName: \"kubernetes.io/projected/bc3156f5-7370-486a-89fb-65087b1f969e-kube-api-access-x2n69\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.686529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.686510    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3156f5-7370-486a-89fb-65087b1f969e-tls-certs\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.686679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.686614    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-tmp-dir\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.686737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.686709    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-home\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.686737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.686724    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-dshm\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.686867 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.686761    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-model-cache\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.687085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.687064    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-tmp-dir\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.689300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.689275    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-dshm\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.689456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.689365    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3156f5-7370-486a-89fb-65087b1f969e-tls-certs\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.698541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.698519    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2n69\" (UniqueName: \"kubernetes.io/projected/bc3156f5-7370-486a-89fb-65087b1f969e-kube-api-access-x2n69\") pod \"test-model-120cb1-kserve-686f649b4f-9w8nf\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") " pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:29.905196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:29.902083    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:30.229143 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:26:30.229100    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc3156f5_7370_486a_89fb_65087b1f969e.slice/crio-1e3c1bd14c33f421867db5514578d79dcb2511f7115274dcc81ca2340bf89150 WatchSource:0}: Error finding container 1e3c1bd14c33f421867db5514578d79dcb2511f7115274dcc81ca2340bf89150: Status 404 returned error can't find the container with id 1e3c1bd14c33f421867db5514578d79dcb2511f7115274dcc81ca2340bf89150
Oct 30 22:26:30.229281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:30.229254    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"]
Oct 30 22:26:30.317294 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:30.317263    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"]
Oct 30 22:26:30.943588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:30.943550    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf" event={"ID":"bc3156f5-7370-486a-89fb-65087b1f969e","Type":"ContainerStarted","Data":"9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53"}
Oct 30 22:26:30.944109 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:30.943597    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf" event={"ID":"bc3156f5-7370-486a-89fb-65087b1f969e","Type":"ContainerStarted","Data":"1e3c1bd14c33f421867db5514578d79dcb2511f7115274dcc81ca2340bf89150"}
Oct 30 22:26:30.966936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:30.966889    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf" podStartSLOduration=1.966873965 podStartE2EDuration="1.966873965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:30.962874667 +0000 UTC m=+2809.620970890" watchObservedRunningTime="2026-10-30 22:26:30.966873965 +0000 UTC m=+2809.624970188"
Oct 30 22:26:31.902814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:31.902779    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:34.652269 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:26:34.652234    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:26:35.317668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:35.317598    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb" podUID="ca26915d-a5a8-4111-aca6-54ee76f8cad9" containerName="payload-pre-processing" containerID="cri-o://848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b" gracePeriod=25
Oct 30 22:26:39.245603 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:39.245030    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"]
Oct 30 22:26:39.245603 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:39.245269    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy" containerID="cri-o://90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c" gracePeriod=30
Oct 30 22:26:39.902762 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:39.902720    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:39.915719 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:39.915692    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:40.041333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:40.041302    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:26:41.412634 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:41.412600    4282 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:26:41.413033 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:41.412653    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:26:44.601592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.601567    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:26:44.672189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.672144    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"]
Oct 30 22:26:44.672577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.672510    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy" containerID="cri-o://bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21" gracePeriod=30
Oct 30 22:26:44.674044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.674011    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-token\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-token\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.674176 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.674072    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istiod-ca-cert\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.674176 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.674127    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-data\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.674308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.674213    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-kube-api-access-2q6q8\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-kube-api-access-2q6q8\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.674376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.674361    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-podinfo\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.675433 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.675402    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istiod-ca-cert" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:44.675569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.675495    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-certs\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.675675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.675653    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-envoy\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.675740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.675714    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-credential-socket\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.675806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.675777    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-socket\") pod \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\" (UID: \"ac8ca6ab-097b-4e0b-a2f1-584306642c62\") "
Oct 30 22:26:44.677265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.677235    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-data" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:44.677654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.677529    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-socket" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:44.677740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.677650    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.677740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.677672    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.679794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.679763    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-credential-socket" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:44.680076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.680042    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-certs" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:44.680581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.680224    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-podinfo" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:44.688233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.687102    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-envoy" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:44.718104 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.718069    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-kube-api-access-2q6q8" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "kube-api-access-2q6q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:44.722483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.722402    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-token" pod "ac8ca6ab-097b-4e0b-a2f1-584306642c62" (UID: "ac8ca6ab-097b-4e0b-a2f1-584306642c62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:44.780296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.780259    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.780296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.780297    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q6q8\" (UniqueName: \"kubernetes.io/projected/ac8ca6ab-097b-4e0b-a2f1-584306642c62-kube-api-access-2q6q8\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.780296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.780314    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.780709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.780324    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.780709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.780333    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.780709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.780343    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.780709 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.780351    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8ca6ab-097b-4e0b-a2f1-584306642c62-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:44.786992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.786962    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb" podUID="ca26915d-a5a8-4111-aca6-54ee76f8cad9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:26:44.981740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.977811    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"]
Oct 30 22:26:44.981740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.979096    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy"
Oct 30 22:26:44.981740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.979118    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy"
Oct 30 22:26:44.981740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.979321    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerName="istio-proxy"
Oct 30 22:26:44.984098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.984075    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:44.991236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.991173    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c92aec\"/\"kube-root-ca.crt\""
Oct 30 22:26:44.991236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.991197    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c92aec\"/\"openshift-service-ca.crt\""
Oct 30 22:26:44.991236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.991209    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c92aec\"/\"default-dockercfg-cddmm\""
Oct 30 22:26:44.992080 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:44.991292    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c92aec\"/\"test-model-c92aec-kserve-self-signed-certs\""
Oct 30 22:26:45.000947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.000922    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"]
Oct 30 22:26:45.072132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.072099    4282 generic.go:356] "Generic (PLEG): container finished" podID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" containerID="90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c" exitCode=0
Oct 30 22:26:45.072317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.072156    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"
Oct 30 22:26:45.072317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.072172    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" event={"ID":"ac8ca6ab-097b-4e0b-a2f1-584306642c62","Type":"ContainerDied","Data":"90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c"}
Oct 30 22:26:45.072317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.072221    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2" event={"ID":"ac8ca6ab-097b-4e0b-a2f1-584306642c62","Type":"ContainerDied","Data":"bf02ce0d3c29e06bd17d122cf021748cc556aea0d4326082fb792afa7f6d53ad"}
Oct 30 22:26:45.072317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.072244    4282 scope.go:122] "RemoveContainer" containerID="90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c"
Oct 30 22:26:45.085141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.085113    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-dshm\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.085258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.085144    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb55g\" (UniqueName: \"kubernetes.io/projected/a002efa8-1326-4799-b1a9-c479bec25a4d-kube-api-access-cb55g\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.085417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.085325    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-home\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.085417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.085337    4282 scope.go:122] "RemoveContainer" containerID="90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c"
Oct 30 22:26:45.085417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.085352    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-model-cache\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.085417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.085380    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a002efa8-1326-4799-b1a9-c479bec25a4d-tls-certs\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.085645 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.085579    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-tmp-dir\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.085740 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:26:45.085712    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c\": container with ID starting with 90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c not found: ID does not exist" containerID="90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c"
Oct 30 22:26:45.085784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.085753    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c"} err="failed to get container status \"90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c\": rpc error: code = NotFound desc = could not find container \"90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c\": container with ID starting with 90a98a1611cd436c549d93340b3feaa874fd362416e2cddc3327f803aaf2b52c not found: ID does not exist"
Oct 30 22:26:45.100555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.100533    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"]
Oct 30 22:26:45.107672 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.107651    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-nw5x2"]
Oct 30 22:26:45.186731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.186701    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-tmp-dir\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.186893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.186822    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-dshm\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.186893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.186841    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb55g\" (UniqueName: \"kubernetes.io/projected/a002efa8-1326-4799-b1a9-c479bec25a4d-kube-api-access-cb55g\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.187012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.186969    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-home\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.187012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.186991    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-model-cache\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.187155 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.187019    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a002efa8-1326-4799-b1a9-c479bec25a4d-tls-certs\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.187155 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.187133    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-tmp-dir\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.187384 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.187358    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-model-cache\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.187548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.187433    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-home\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.189287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.189269    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-dshm\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.189707 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.189689    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a002efa8-1326-4799-b1a9-c479bec25a4d-tls-certs\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.233312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.233191    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb55g\" (UniqueName: \"kubernetes.io/projected/a002efa8-1326-4799-b1a9-c479bec25a4d-kube-api-access-cb55g\") pod \"test-model-c92aec-kserve-5765d74789-6rxm8\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") " pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.309514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.309245    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:45.472014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.471980    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8ca6ab-097b-4e0b-a2f1-584306642c62" path="/var/lib/kubelet/pods/ac8ca6ab-097b-4e0b-a2f1-584306642c62/volumes"
Oct 30 22:26:45.648829 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:26:45.648791    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda002efa8_1326_4799_b1a9_c479bec25a4d.slice/crio-b315b8376d155ecfe0a04dcc7496c2bd902c7347dae9559dc1bf0d242243416b WatchSource:0}: Error finding container b315b8376d155ecfe0a04dcc7496c2bd902c7347dae9559dc1bf0d242243416b: Status 404 returned error can't find the container with id b315b8376d155ecfe0a04dcc7496c2bd902c7347dae9559dc1bf0d242243416b
Oct 30 22:26:45.649171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:45.649092    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"]
Oct 30 22:26:46.083207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:46.083177    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8" event={"ID":"a002efa8-1326-4799-b1a9-c479bec25a4d","Type":"ContainerStarted","Data":"c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f"}
Oct 30 22:26:46.083417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:46.083215    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8" event={"ID":"a002efa8-1326-4799-b1a9-c479bec25a4d","Type":"ContainerStarted","Data":"b315b8376d155ecfe0a04dcc7496c2bd902c7347dae9559dc1bf0d242243416b"}
Oct 30 22:26:46.109462 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:46.109401    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8" podStartSLOduration=2.109387096 podStartE2EDuration="2.109387096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:46.10641164 +0000 UTC m=+2824.764507864" watchObservedRunningTime="2026-10-30 22:26:46.109387096 +0000 UTC m=+2824.767483310"
Oct 30 22:26:46.310140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:46.310109    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:49.971858 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:49.971832    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 22:26:50.097206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.097171    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-socket\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.097428 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.097232    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-kube-api-access-xbhwp\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-kube-api-access-xbhwp\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.097428 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.097278    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-podinfo\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.097726 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.097429    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940dbb6d-1424-4a15-89df-a9ec9d77534d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940dbb6d-1424-4a15-89df-a9ec9d77534d-istiod-ca-cert\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.097811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.097728    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-envoy\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.097880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.097855    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-data\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.097963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.097928    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-token\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-token\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.098027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.098005    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-credential-socket\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.098085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.098074    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-certs\") pod \"940dbb6d-1424-4a15-89df-a9ec9d77534d\" (UID: \"940dbb6d-1424-4a15-89df-a9ec9d77534d\") "
Oct 30 22:26:50.098549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.098461    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-socket" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.098732 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.098683    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-data" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.098859 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.098743    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-certs" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.098929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.098908    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-credential-socket" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.098989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.098977    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.099035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.098997    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.099035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.099009    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.099432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.099411    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940dbb6d-1424-4a15-89df-a9ec9d77534d-istiod-ca-cert" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:50.101118 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.101091    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-kube-api-access-xbhwp" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "kube-api-access-xbhwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:50.101233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.101197    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-podinfo" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:26:50.101233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.101205    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-envoy" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:50.101577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.101555    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-token" pod "940dbb6d-1424-4a15-89df-a9ec9d77534d" (UID: "940dbb6d-1424-4a15-89df-a9ec9d77534d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:50.119819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.119785    4282 generic.go:356] "Generic (PLEG): container finished" podID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerID="bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21" exitCode=0
Oct 30 22:26:50.119962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.119854    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"
Oct 30 22:26:50.119962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.119893    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" event={"ID":"940dbb6d-1424-4a15-89df-a9ec9d77534d","Type":"ContainerDied","Data":"bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21"}
Oct 30 22:26:50.119962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.119959    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" event={"ID":"940dbb6d-1424-4a15-89df-a9ec9d77534d","Type":"ContainerDied","Data":"00f924ecebeb129722546c25f02f54b1386ac4526ee8ee093c02dec1b236e1df"}
Oct 30 22:26:50.120158 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.120006    4282 scope.go:122] "RemoveContainer" containerID="bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21"
Oct 30 22:26:50.143744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.143726    4282 scope.go:122] "RemoveContainer" containerID="bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21"
Oct 30 22:26:50.144049 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:26:50.144028    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21\": container with ID starting with bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21 not found: ID does not exist" containerID="bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21"
Oct 30 22:26:50.144094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.144058    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21"} err="failed to get container status \"bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21\": rpc error: code = NotFound desc = could not find container \"bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21\": container with ID starting with bcae19932644217053c71d5fd434e8e65514baac468a773ad54a798121886e21 not found: ID does not exist"
Oct 30 22:26:50.166676 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.166652    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"]
Oct 30 22:26:50.181658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.181624    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl"]
Oct 30 22:26:50.200350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.200286    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.200350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.200311    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940dbb6d-1424-4a15-89df-a9ec9d77534d-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.200350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.200322    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.200350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.200330    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.200350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.200339    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940dbb6d-1424-4a15-89df-a9ec9d77534d-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.200350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.200347    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhwp\" (UniqueName: \"kubernetes.io/projected/940dbb6d-1424-4a15-89df-a9ec9d77534d-kube-api-access-xbhwp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:50.916202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.916172    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:26:50.916353 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:50.916236    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-fm4vl" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:26:51.465856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:51.465825    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" path="/var/lib/kubelet/pods/940dbb6d-1424-4a15-89df-a9ec9d77534d/volumes"
Oct 30 22:26:53.799740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.799666    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"]
Oct 30 22:26:53.801116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.801093    4282 cpu_manager.go:414] "containerMap: removing container" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy"
Oct 30 22:26:53.801170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.801121    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy"
Oct 30 22:26:53.801318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.801304    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="940dbb6d-1424-4a15-89df-a9ec9d77534d" containerName="istio-proxy"
Oct 30 22:26:53.807128 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.807105    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:53.816136 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.816114    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28d39b\"/\"kube-root-ca.crt\""
Oct 30 22:26:53.816333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.816191    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28d39b\"/\"default-dockercfg-kgxhr\""
Oct 30 22:26:53.816333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.816194    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28d39b\"/\"test-model-28d39b-kserve-self-signed-certs\""
Oct 30 22:26:53.816585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.816243    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28d39b\"/\"openshift-service-ca.crt\""
Oct 30 22:26:53.818451 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:53.818426    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"]
Oct 30 22:26:54.000692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.000654    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5z6k\" (UniqueName: \"kubernetes.io/projected/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-kube-api-access-m5z6k\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.015205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.015168    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-model-cache\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.015547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.015528    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tmp-dir\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.015788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.015768    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-dshm\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.016121 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.016102    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tls-certs\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.016320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.016305    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-home\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.118169    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tls-certs\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.118253    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-home\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.118441    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5z6k\" (UniqueName: \"kubernetes.io/projected/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-kube-api-access-m5z6k\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.118502    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-model-cache\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.118528    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tmp-dir\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.118593    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-dshm\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.120201    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-home\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.120758    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tmp-dir\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.122540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.121323    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-model-cache\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.137790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.135452    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5z6k\" (UniqueName: \"kubernetes.io/projected/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-kube-api-access-m5z6k\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.142398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.142369    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tls-certs\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.150000 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.149967    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-dshm\") pod \"test-model-28d39b-kserve-587775484c-vg46m\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") " pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.423705 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.423673    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:54.728689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.728647    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"]
Oct 30 22:26:54.728689 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:26:54.728647    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41e3b9df_e8c1_45f9_8e01_baa67142e7ca.slice/crio-2bef9fb9edf37f449fac30df079a527887f50355d1aa8fc5190372ff0d095e0e WatchSource:0}: Error finding container 2bef9fb9edf37f449fac30df079a527887f50355d1aa8fc5190372ff0d095e0e: Status 404 returned error can't find the container with id 2bef9fb9edf37f449fac30df079a527887f50355d1aa8fc5190372ff0d095e0e
Oct 30 22:26:54.787198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:54.787161    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb" podUID="ca26915d-a5a8-4111-aca6-54ee76f8cad9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:26:55.170949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.170912    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m" event={"ID":"41e3b9df-e8c1-45f9-8e01-baa67142e7ca","Type":"ContainerStarted","Data":"98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c"}
Oct 30 22:26:55.171489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.170964    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m" event={"ID":"41e3b9df-e8c1-45f9-8e01-baa67142e7ca","Type":"ContainerStarted","Data":"2bef9fb9edf37f449fac30df079a527887f50355d1aa8fc5190372ff0d095e0e"}
Oct 30 22:26:55.195374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.195326    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m" podStartSLOduration=2.195312849 podStartE2EDuration="2.195312849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:55.192308924 +0000 UTC m=+2833.850405146" watchObservedRunningTime="2026-10-30 22:26:55.195312849 +0000 UTC m=+2833.853409071"
Oct 30 22:26:55.243979 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.243951    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"]
Oct 30 22:26:55.244406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.244364    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf" podUID="5dc97cfc-917f-4a48-8651-ef29ebaf4b56" containerName="maas-api" containerID="cri-o://b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d" gracePeriod=30
Oct 30 22:26:55.310848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.310821    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:55.324045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.324016    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:55.423837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.423774    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:26:55.452014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.447916    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf" podUID="5dc97cfc-917f-4a48-8651-ef29ebaf4b56" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.171:8443/health\": dial tcp 10.133.0.171:8443: connect: connection refused"
Oct 30 22:26:55.993969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:55.993934    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:26:56.018546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.018520    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:26:56.175404 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.175372    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-openshift-service-ca\") pod \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") "
Oct 30 22:26:56.175844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.175534    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca26915d-a5a8-4111-aca6-54ee76f8cad9-kube-api-access-rbdm2\" (UniqueName: \"kubernetes.io/projected/ca26915d-a5a8-4111-aca6-54ee76f8cad9-kube-api-access-rbdm2\") pod \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\" (UID: \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\") "
Oct 30 22:26:56.175844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.175560    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls\") pod \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") "
Oct 30 22:26:56.175844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.175650    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca26915d-a5a8-4111-aca6-54ee76f8cad9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca26915d-a5a8-4111-aca6-54ee76f8cad9-plugins-config-volume\") pod \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\" (UID: \"ca26915d-a5a8-4111-aca6-54ee76f8cad9\") "
Oct 30 22:26:56.175844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.175699    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-metrics-certs\") pod \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") "
Oct 30 22:26:56.175844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.175734    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-openshift-service-ca" pod "5dc97cfc-917f-4a48-8651-ef29ebaf4b56" (UID: "5dc97cfc-917f-4a48-8651-ef29ebaf4b56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:56.176142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.175882    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-kube-api-access-7wng4\" (UniqueName: \"kubernetes.io/projected/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-kube-api-access-7wng4\") pod \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\" (UID: \"5dc97cfc-917f-4a48-8651-ef29ebaf4b56\") "
Oct 30 22:26:56.176142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.176033    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca26915d-a5a8-4111-aca6-54ee76f8cad9-plugins-config-volume" pod "ca26915d-a5a8-4111-aca6-54ee76f8cad9" (UID: "ca26915d-a5a8-4111-aca6-54ee76f8cad9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:56.176714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.176689    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca26915d-a5a8-4111-aca6-54ee76f8cad9-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:56.176869 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.176717    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:56.178562 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.178534    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-metrics-certs" pod "5dc97cfc-917f-4a48-8651-ef29ebaf4b56" (UID: "5dc97cfc-917f-4a48-8651-ef29ebaf4b56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:56.178680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.178552    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls" pod "5dc97cfc-917f-4a48-8651-ef29ebaf4b56" (UID: "5dc97cfc-917f-4a48-8651-ef29ebaf4b56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:56.178746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.178708    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca26915d-a5a8-4111-aca6-54ee76f8cad9-kube-api-access-rbdm2" pod "ca26915d-a5a8-4111-aca6-54ee76f8cad9" (UID: "ca26915d-a5a8-4111-aca6-54ee76f8cad9"). InnerVolumeSpecName "kube-api-access-rbdm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:56.178874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.178854    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-kube-api-access-7wng4" pod "5dc97cfc-917f-4a48-8651-ef29ebaf4b56" (UID: "5dc97cfc-917f-4a48-8651-ef29ebaf4b56"). InnerVolumeSpecName "kube-api-access-7wng4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:56.181451 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.181425    4282 generic.go:356] "Generic (PLEG): container finished" podID="5dc97cfc-917f-4a48-8651-ef29ebaf4b56" containerID="b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d" exitCode=0
Oct 30 22:26:56.181612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.181503    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"
Oct 30 22:26:56.181612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.181499    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf" event={"ID":"5dc97cfc-917f-4a48-8651-ef29ebaf4b56","Type":"ContainerDied","Data":"b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d"}
Oct 30 22:26:56.181612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.181561    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf" event={"ID":"5dc97cfc-917f-4a48-8651-ef29ebaf4b56","Type":"ContainerDied","Data":"27e2aa0a3c4a9f0feab6662f7945381d48f4af75d3763f2b1a78d58f1e7611dc"}
Oct 30 22:26:56.181612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.181584    4282 scope.go:122] "RemoveContainer" containerID="b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d"
Oct 30 22:26:56.183242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.183219    4282 generic.go:356] "Generic (PLEG): container finished" podID="ca26915d-a5a8-4111-aca6-54ee76f8cad9" containerID="848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b" exitCode=0
Oct 30 22:26:56.183339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.183290    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb" event={"ID":"ca26915d-a5a8-4111-aca6-54ee76f8cad9","Type":"ContainerDied","Data":"848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b"}
Oct 30 22:26:56.183399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.183344    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb" event={"ID":"ca26915d-a5a8-4111-aca6-54ee76f8cad9","Type":"ContainerDied","Data":"804ba01c582d3bbfbd797b602d2646c9b651111bb2b8b875339bc640df5fec86"}
Oct 30 22:26:56.183524 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.183490    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"
Oct 30 22:26:56.197239 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.197220    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:26:56.217868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.217851    4282 scope.go:122] "RemoveContainer" containerID="b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d"
Oct 30 22:26:56.218212 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:26:56.218192    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d\": container with ID starting with b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d not found: ID does not exist" containerID="b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d"
Oct 30 22:26:56.218312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.218222    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d"} err="failed to get container status \"b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d\": rpc error: code = NotFound desc = could not find container \"b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d\": container with ID starting with b5844cb3e63cf6c0bffdaae93bbf19ec4bcf40992d1091ff0e2f1912936c5d9d not found: ID does not exist"
Oct 30 22:26:56.218312 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.218240    4282 scope.go:122] "RemoveContainer" containerID="848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b"
Oct 30 22:26:56.231547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.231440    4282 scope.go:122] "RemoveContainer" containerID="848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b"
Oct 30 22:26:56.231794 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:26:56.231774    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b\": container with ID starting with 848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b not found: ID does not exist" containerID="848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b"
Oct 30 22:26:56.231897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.231803    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b"} err="failed to get container status \"848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b\": rpc error: code = NotFound desc = could not find container \"848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b\": container with ID starting with 848927c146f581b772c009d54f2a835cc43f9c7254b36703b031573f776e661b not found: ID does not exist"
Oct 30 22:26:56.246076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.246048    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"]
Oct 30 22:26:56.252273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.252249    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-9c8ecf-96dd6c4bd-b7rpf"]
Oct 30 22:26:56.264567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.264546    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"]
Oct 30 22:26:56.269740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.269716    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5313f0-65d6dc65cf-x2mfb"]
Oct 30 22:26:56.278798 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.278768    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wng4\" (UniqueName: \"kubernetes.io/projected/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-kube-api-access-7wng4\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:56.278927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.278866    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbdm2\" (UniqueName: \"kubernetes.io/projected/ca26915d-a5a8-4111-aca6-54ee76f8cad9-kube-api-access-rbdm2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:56.278927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.278913    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:56.279032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:56.278982    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc97cfc-917f-4a48-8651-ef29ebaf4b56-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:57.466544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.466511    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc97cfc-917f-4a48-8651-ef29ebaf4b56" path="/var/lib/kubelet/pods/5dc97cfc-917f-4a48-8651-ef29ebaf4b56/volumes"
Oct 30 22:26:57.467017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.467002    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca26915d-a5a8-4111-aca6-54ee76f8cad9" path="/var/lib/kubelet/pods/ca26915d-a5a8-4111-aca6-54ee76f8cad9/volumes"
Oct 30 22:26:57.934105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.934069    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"]
Oct 30 22:26:57.934952 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.934930    4282 cpu_manager.go:414] "containerMap: removing container" podUID="5dc97cfc-917f-4a48-8651-ef29ebaf4b56" containerName="maas-api"
Oct 30 22:26:57.934952 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.934952    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc97cfc-917f-4a48-8651-ef29ebaf4b56" containerName="maas-api"
Oct 30 22:26:57.935147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.934997    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ca26915d-a5a8-4111-aca6-54ee76f8cad9" containerName="payload-pre-processing"
Oct 30 22:26:57.935147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.935006    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca26915d-a5a8-4111-aca6-54ee76f8cad9" containerName="payload-pre-processing"
Oct 30 22:26:57.935261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.935221    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca26915d-a5a8-4111-aca6-54ee76f8cad9" containerName="payload-pre-processing"
Oct 30 22:26:57.935261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.935247    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc97cfc-917f-4a48-8651-ef29ebaf4b56" containerName="maas-api"
Oct 30 22:26:57.940646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.940625    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:57.958023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:57.958001    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"]
Oct 30 22:26:58.030201 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030163    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.030201 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030215    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.032969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030240    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.032969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030405    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.032969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030507    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.032969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030657    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.032969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030745    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.032969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030820    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72fz7\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-kube-api-access-72fz7\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.032969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.030976    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.132805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.132774    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.132990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.132868    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.132990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.132891    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.132990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.132905    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.132990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.132967    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.133246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.133009    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.133246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.133062    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.133246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.133123    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.133246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.133201    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72fz7\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-kube-api-access-72fz7\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.133539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.133507    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.133671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.133639    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.133910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.133701    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.133910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.133859    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.134116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.134096    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.135512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.135456    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.135713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.135694    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.144988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.144960    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.145230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.145210    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72fz7\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-kube-api-access-72fz7\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.256699 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.256615    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:58.554936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.554892    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"]
Oct 30 22:26:58.555388 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:26:58.555087    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc6b7c6_166a_4d5f_8da3_ee7345189759.slice/crio-3568f359bf74ab7aff96ecc920943333dfd5d1a48a388619e9fb88830ec6143c WatchSource:0}: Error finding container 3568f359bf74ab7aff96ecc920943333dfd5d1a48a388619e9fb88830ec6143c: Status 404 returned error can't find the container with id 3568f359bf74ab7aff96ecc920943333dfd5d1a48a388619e9fb88830ec6143c
Oct 30 22:26:58.558325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.558294    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:58.558459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.558360    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:58.558459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:58.558391    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:26:59.219236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:59.219188    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" event={"ID":"5fc6b7c6-166a-4d5f-8da3-ee7345189759","Type":"ContainerStarted","Data":"efd2e4ae86e0db3cf23d0bfb8c41f65653965a3a13d0afdaf95d5ecdefb39c83"}
Oct 30 22:26:59.219448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:59.219352    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" event={"ID":"5fc6b7c6-166a-4d5f-8da3-ee7345189759","Type":"ContainerStarted","Data":"3568f359bf74ab7aff96ecc920943333dfd5d1a48a388619e9fb88830ec6143c"}
Oct 30 22:26:59.250703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:59.250642    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podStartSLOduration=2.250623397 podStartE2EDuration="2.250623397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:59.244426528 +0000 UTC m=+2837.902522748" watchObservedRunningTime="2026-10-30 22:26:59.250623397 +0000 UTC m=+2837.908719621"
Oct 30 22:26:59.257178 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:59.257155    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:26:59.258601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:59.258582    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 22:26:59.258654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:26:59.258636    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 22:27:00.276551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:00.269917    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 22:27:00.276551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:00.269995    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 22:27:01.277606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:01.267835    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 22:27:01.277606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:01.267890    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 22:27:02.271493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:02.270581    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 22:27:02.271493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:02.270666    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 22:27:03.274494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:03.272676    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 22:27:03.274494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:03.272737    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 22:27:04.090803 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:04.090766    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"]
Oct 30 22:27:04.091047 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:04.091020    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy" containerID="cri-o://f1003e352c6b1a734a3321474142f9eab9936c01c196b367f3c1bf601d7bff93" gracePeriod=30
Oct 30 22:27:04.258582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:04.257837    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 22:27:04.258582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:04.257917    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 22:27:04.434492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:04.430653    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:27:04.549838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:04.549813    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:27:04.680532 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:27:04.679539    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:27:05.257608 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:05.257566    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body=
Oct 30 22:27:05.257853 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:05.257628    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused"
Oct 30 22:27:05.359227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:05.359193    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:27:06.262291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:06.262264    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:27:06.262765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:06.262660    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:27:06.263284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:06.263260    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:27:09.385646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.385609    4282 generic.go:356] "Generic (PLEG): container finished" podID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerID="f1003e352c6b1a734a3321474142f9eab9936c01c196b367f3c1bf601d7bff93" exitCode=0
Oct 30 22:27:09.385986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.385689    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" event={"ID":"6cd7a3fd-f269-4d19-a42a-0899b8e8104c","Type":"ContainerDied","Data":"f1003e352c6b1a734a3321474142f9eab9936c01c196b367f3c1bf601d7bff93"}
Oct 30 22:27:09.385986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.385753    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd" event={"ID":"6cd7a3fd-f269-4d19-a42a-0899b8e8104c","Type":"ContainerDied","Data":"c7c3ec58290f6f9191fdc9af2b9c6d6a9134e624704b26246227646499cd3008"}
Oct 30 22:27:09.385986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.385776    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7c3ec58290f6f9191fdc9af2b9c6d6a9134e624704b26246227646499cd3008"
Oct 30 22:27:09.409862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.409814    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:27:09.526302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526272    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-socket\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.526618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526385    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-data\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.526618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526547    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-envoy\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.526618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526593    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-credential-socket\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.526845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526629    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istiod-ca-cert\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.526845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526694    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-certs\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.526845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526720    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-kube-api-access-t8wrq\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-kube-api-access-t8wrq\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.526845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526775    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-token\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-token\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.526845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526817    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-podinfo\") pod \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\" (UID: \"6cd7a3fd-f269-4d19-a42a-0899b8e8104c\") "
Oct 30 22:27:09.527509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526615    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-socket" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:09.527681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.526701    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-data" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:09.527772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.527042    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istiod-ca-cert" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:09.527839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.527281    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-credential-socket" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:09.527839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.527443    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-certs" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:09.529930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.529894    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-podinfo" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:09.539545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.539127    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"]
Oct 30 22:27:09.541725 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.541700    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-envoy" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:09.543763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.543741    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-kube-api-access-t8wrq" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "kube-api-access-t8wrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:09.544430 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.544398    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-token" pod "6cd7a3fd-f269-4d19-a42a-0899b8e8104c" (UID: "6cd7a3fd-f269-4d19-a42a-0899b8e8104c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:09.629893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629818    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.629893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629846    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.629893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629856    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.629893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629867    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wrq\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-kube-api-access-t8wrq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.629893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629878    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.629893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629885    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.629893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629895    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.630243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629903    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.630243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:09.629911    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd7a3fd-f269-4d19-a42a-0899b8e8104c-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:10.393060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:10.393030    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"
Oct 30 22:27:10.423220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:10.423186    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"]
Oct 30 22:27:10.430833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:10.430803    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-9c8ecf-openshift-default-858f6c44fc-nggrd"]
Oct 30 22:27:11.467396 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:11.467360    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" path="/var/lib/kubelet/pods/6cd7a3fd-f269-4d19-a42a-0899b8e8104c/volumes"
Oct 30 22:27:24.565197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.565158    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main" containerID="cri-o://7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331" gracePeriod=45
Oct 30 22:27:24.907167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.906833    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:27:24.951733 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.951691    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-model-cache\") pod \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") "
Oct 30 22:27:24.952059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.951882    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-home\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-home\") pod \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") "
Oct 30 22:27:24.952059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.951916    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tls-certs\" (UniqueName: \"kubernetes.io/secret/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tls-certs\") pod \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") "
Oct 30 22:27:24.952059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.952025    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-model-cache" pod "71a166e3-0bc4-4f50-af2a-a8b342fd9421" (UID: "71a166e3-0bc4-4f50-af2a-a8b342fd9421"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:24.952283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.952129    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-dshm\") pod \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") "
Oct 30 22:27:24.952283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.952237    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a166e3-0bc4-4f50-af2a-a8b342fd9421-kube-api-access-n8756\" (UniqueName: \"kubernetes.io/projected/71a166e3-0bc4-4f50-af2a-a8b342fd9421-kube-api-access-n8756\") pod \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") "
Oct 30 22:27:24.952283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.952241    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-home" pod "71a166e3-0bc4-4f50-af2a-a8b342fd9421" (UID: "71a166e3-0bc4-4f50-af2a-a8b342fd9421"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:24.952592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.952282    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tmp-dir\") pod \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\" (UID: \"71a166e3-0bc4-4f50-af2a-a8b342fd9421\") "
Oct 30 22:27:24.952927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.952902    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tmp-dir" pod "71a166e3-0bc4-4f50-af2a-a8b342fd9421" (UID: "71a166e3-0bc4-4f50-af2a-a8b342fd9421"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:24.953196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.953137    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:24.953196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.953161    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:24.953196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.953176    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:24.955660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.955461    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tls-certs" pod "71a166e3-0bc4-4f50-af2a-a8b342fd9421" (UID: "71a166e3-0bc4-4f50-af2a-a8b342fd9421"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:24.956830 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.956795    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a166e3-0bc4-4f50-af2a-a8b342fd9421-kube-api-access-n8756" pod "71a166e3-0bc4-4f50-af2a-a8b342fd9421" (UID: "71a166e3-0bc4-4f50-af2a-a8b342fd9421"). InnerVolumeSpecName "kube-api-access-n8756". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:24.956924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:24.956882    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-dshm" pod "71a166e3-0bc4-4f50-af2a-a8b342fd9421" (UID: "71a166e3-0bc4-4f50-af2a-a8b342fd9421"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:25.054544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.054513    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71a166e3-0bc4-4f50-af2a-a8b342fd9421-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:25.054544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.054544    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8756\" (UniqueName: \"kubernetes.io/projected/71a166e3-0bc4-4f50-af2a-a8b342fd9421-kube-api-access-n8756\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:25.054843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.054556    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71a166e3-0bc4-4f50-af2a-a8b342fd9421-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:25.557172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.557143    4282 generic.go:356] "Generic (PLEG): container finished" podID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerID="7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331" exitCode=0
Oct 30 22:27:25.557363 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.557189    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"
Oct 30 22:27:25.557363 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.557229    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" event={"ID":"71a166e3-0bc4-4f50-af2a-a8b342fd9421","Type":"ContainerDied","Data":"7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331"}
Oct 30 22:27:25.557363 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.557275    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7" event={"ID":"71a166e3-0bc4-4f50-af2a-a8b342fd9421","Type":"ContainerDied","Data":"343bca23282b2ac78ccccfcc18aedda8d5e103347830881a15380ef21615fb0b"}
Oct 30 22:27:25.557363 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.557294    4282 scope.go:122] "RemoveContainer" containerID="7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331"
Oct 30 22:27:25.569747 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.569518    4282 scope.go:122] "RemoveContainer" containerID="7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331"
Oct 30 22:27:25.570040 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:27:25.569832    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331\": container with ID starting with 7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331 not found: ID does not exist" containerID="7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331"
Oct 30 22:27:25.570040 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.569859    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331"} err="failed to get container status \"7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331\": rpc error: code = NotFound desc = could not find container \"7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331\": container with ID starting with 7d478225a32910bb90bc61ed79afc9750f6bc34e810b11fef235ac992ed8e331 not found: ID does not exist"
Oct 30 22:27:25.575715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.575692    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"]
Oct 30 22:27:25.580779 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:25.580760    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b2e72/distinct-2-w2-2b2e72-kserve-8694898d5b-bjjg7"]
Oct 30 22:27:26.376425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:26.376387    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:27:27.465976 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:27.465947    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" path="/var/lib/kubelet/pods/71a166e3-0bc4-4f50-af2a-a8b342fd9421/volumes"
Oct 30 22:27:34.651161 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:27:34.651123    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:27:43.008311 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.008269    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"]
Oct 30 22:27:43.009357 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.009332    4282 cpu_manager.go:414] "containerMap: removing container" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy"
Oct 30 22:27:43.009498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.009358    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy"
Oct 30 22:27:43.009498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.009392    4282 cpu_manager.go:414] "containerMap: removing container" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main"
Oct 30 22:27:43.009498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.009400    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main"
Oct 30 22:27:43.009715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.009654    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a166e3-0bc4-4f50-af2a-a8b342fd9421" containerName="main"
Oct 30 22:27:43.009715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.009690    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd7a3fd-f269-4d19-a42a-0899b8e8104c" containerName="istio-proxy"
Oct 30 22:27:43.013659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.013638    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.026365 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.026339    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"]
Oct 30 22:27:43.076448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.076408    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.076651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.076598    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.076827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.076798    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.076896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.076881    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.076955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.076921    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.077027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.077010    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.077090 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.077044    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.077090 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.077071    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.077213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.077191    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzm7m\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-kube-api-access-nzm7m\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.178874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.178843    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.178905    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.178938    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179080    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179117    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179214 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179135    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179237    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzm7m\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-kube-api-access-nzm7m\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179392    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179517    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179529    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179518    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179628    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.179750 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.179729    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.180093 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.180074    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.181628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.181605    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.181758 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.181738    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.190142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.190121    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.190615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.190594    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzm7m\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-kube-api-access-nzm7m\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.334647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.334550    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:43.675011 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.674983    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"]
Oct 30 22:27:43.675249 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:27:43.675219    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e43d9d_1cf7_418f_a158_1a2d002dafcc.slice/crio-6732646f8e2f707dd641bd52c64b03cfbf07952f677ed9c4a579c1c7d8e135e3 WatchSource:0}: Error finding container 6732646f8e2f707dd641bd52c64b03cfbf07952f677ed9c4a579c1c7d8e135e3: Status 404 returned error can't find the container with id 6732646f8e2f707dd641bd52c64b03cfbf07952f677ed9c4a579c1c7d8e135e3
Oct 30 22:27:43.678674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.678641    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:43.678738 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.678707    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:43.678776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.678739    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:43.786014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.785977    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" event={"ID":"d3e43d9d-1cf7-418f-a158-1a2d002dafcc","Type":"ContainerStarted","Data":"a4c047c8c6f94589b346f287701803612dbad1074e6910ed4496d5602eb6eaec"}
Oct 30 22:27:43.786195 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.786035    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" event={"ID":"d3e43d9d-1cf7-418f-a158-1a2d002dafcc","Type":"ContainerStarted","Data":"6732646f8e2f707dd641bd52c64b03cfbf07952f677ed9c4a579c1c7d8e135e3"}
Oct 30 22:27:43.811367 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:43.811312    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podStartSLOduration=1.811298262 podStartE2EDuration="1.811298262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:43.807934746 +0000 UTC m=+2882.466030970" watchObservedRunningTime="2026-10-30 22:27:43.811298262 +0000 UTC m=+2882.469394485"
Oct 30 22:27:44.349181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:44.349151    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:44.350956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:44.350927    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 22:27:44.351096 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:44.350978    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 22:27:45.349406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:45.349373    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 22:27:45.349837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:45.349425    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 22:27:45.842806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:45.842779    4282 scope.go:122] "RemoveContainer" containerID="f1003e352c6b1a734a3321474142f9eab9936c01c196b367f3c1bf601d7bff93"
Oct 30 22:27:46.348675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:46.348640    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 22:27:46.348990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:46.348957    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 22:27:47.348779 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:47.348743    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 22:27:47.349202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:47.348806    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 22:27:48.349444 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.349409    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 22:27:48.349956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.349523    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 22:27:48.580862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.580812    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"]
Oct 30 22:27:48.587976 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.587948    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"]
Oct 30 22:27:48.588146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.588063    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.591596 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.591572    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-28d39b-openshift-default-dockercfg-qsmx6\""
Oct 30 22:27:48.712405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.712374    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.712648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.712537    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.712648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.712585    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.712780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.712681    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.712838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.712776    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a223810-9530-490c-8cfe-82e27f55b98b-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.712894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.712834    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.712894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.712874    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qkr\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-kube-api-access-j2qkr\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.713005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.712982    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.713067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.713053    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a223810-9530-490c-8cfe-82e27f55b98b-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814289 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814256    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a223810-9530-490c-8cfe-82e27f55b98b-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814319    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814342    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qkr\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-kube-api-access-j2qkr\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814553    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814655    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a223810-9530-490c-8cfe-82e27f55b98b-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814752    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814866 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814854    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814898    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.814930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814910    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.815040 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.814947    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.815205 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.815183    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.815332 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.815286    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.815387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.815376    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.815630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.815602    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a223810-9530-490c-8cfe-82e27f55b98b-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.817282 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.817260    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a223810-9530-490c-8cfe-82e27f55b98b-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.817366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.817332    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.829562 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.829538    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qkr\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-kube-api-access-j2qkr\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.834977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.834956    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:48.914542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:48.912791    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:49.286264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.286118    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"]
Oct 30 22:27:49.286566 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:27:49.286516    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a223810_9530_490c_8cfe_82e27f55b98b.slice/crio-234265c1b0a3fa8c6d298d814fc297c199dd1e600477e97d601aa44a0e3721c2 WatchSource:0}: Error finding container 234265c1b0a3fa8c6d298d814fc297c199dd1e600477e97d601aa44a0e3721c2: Status 404 returned error can't find the container with id 234265c1b0a3fa8c6d298d814fc297c199dd1e600477e97d601aa44a0e3721c2
Oct 30 22:27:49.289876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.289843    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:49.289945 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.289912    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:49.289945 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.289942    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:49.349296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.349271    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 22:27:49.349400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.349319    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 22:27:49.492325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.492284    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"]
Oct 30 22:27:49.497259 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.497232    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.504577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.504552    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"]
Oct 30 22:27:49.510170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.510146    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c92aec-openshift-default-dockercfg-jtz49\""
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680061    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680143    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4440c1-907a-4752-9a69-197dfcb5ead4-istiod-ca-cert\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680196    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-podinfo\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680383    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-data\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680508    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-token\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680734    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-credential-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680790    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-certs\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680843    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-envoy\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.690495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.680871    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldd79\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-kube-api-access-ldd79\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.782510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.782453    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-data\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.782715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.782556    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-token\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.782803 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.782737    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-credential-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.782803 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.782786    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-certs\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.782917 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.782832    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-envoy\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.782917 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.782862    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldd79\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-kube-api-access-ldd79\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.782917 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.782889    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-data\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.783089 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.782960    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.783089 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.783022    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4440c1-907a-4752-9a69-197dfcb5ead4-istiod-ca-cert\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.783089 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.783073    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-podinfo\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.783258 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.783181    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-certs\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.783567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.783541    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-credential-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.783711 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.783548    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-socket\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.783773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.783733    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4440c1-907a-4752-9a69-197dfcb5ead4-istiod-ca-cert\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.785716 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.785694    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-envoy\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.786022 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.786004    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-podinfo\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.794096 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.794069    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-token\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.794261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.794238    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldd79\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-kube-api-access-ldd79\") pod \"e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") " pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:49.821798 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:49.821769    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:50.023575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.021616    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" event={"ID":"3a223810-9530-490c-8cfe-82e27f55b98b","Type":"ContainerStarted","Data":"7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed"}
Oct 30 22:27:50.023575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.021689    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" event={"ID":"3a223810-9530-490c-8cfe-82e27f55b98b","Type":"ContainerStarted","Data":"234265c1b0a3fa8c6d298d814fc297c199dd1e600477e97d601aa44a0e3721c2"}
Oct 30 22:27:50.071493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.063609    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" podStartSLOduration=2.063589496 podStartE2EDuration="2.063589496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:50.052746691 +0000 UTC m=+2888.710842914" watchObservedRunningTime="2026-10-30 22:27:50.063589496 +0000 UTC m=+2888.721685722"
Oct 30 22:27:50.071493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.067925    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"]
Oct 30 22:27:50.071493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.068110    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5" podUID="26623cb1-5e21-4175-a760-60619be198cf" containerName="maas-api" containerID="cri-o://c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74" gracePeriod=30
Oct 30 22:27:50.200939 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.200592    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"]
Oct 30 22:27:50.208667 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:27:50.208637    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4440c1_907a_4752_9a69_197dfcb5ead4.slice/crio-56fa78b05577860567b5962291ad8ed29eff88cc14d5e017917a70065f32c268 WatchSource:0}: Error finding container 56fa78b05577860567b5962291ad8ed29eff88cc14d5e017917a70065f32c268: Status 404 returned error can't find the container with id 56fa78b05577860567b5962291ad8ed29eff88cc14d5e017917a70065f32c268
Oct 30 22:27:50.214792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.214380    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:50.214792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.214546    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:50.214792 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.214632    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:27:50.350699 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.350669    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body=
Oct 30 22:27:50.351012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.350966    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused"
Oct 30 22:27:50.899513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.899399    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 22:27:50.916199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.913438    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:50.916199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.914545    4282 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:27:50.916199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.914609    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:27:50.933989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.932377    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-maas-api-tls\") pod \"26623cb1-5e21-4175-a760-60619be198cf\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") "
Oct 30 22:27:50.933989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.932670    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26623cb1-5e21-4175-a760-60619be198cf-kube-api-access-cn6d6\" (UniqueName: \"kubernetes.io/projected/26623cb1-5e21-4175-a760-60619be198cf-kube-api-access-cn6d6\") pod \"26623cb1-5e21-4175-a760-60619be198cf\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") "
Oct 30 22:27:50.933989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.932823    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26623cb1-5e21-4175-a760-60619be198cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26623cb1-5e21-4175-a760-60619be198cf-openshift-service-ca\") pod \"26623cb1-5e21-4175-a760-60619be198cf\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") "
Oct 30 22:27:50.933989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.932857    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-metrics-certs\") pod \"26623cb1-5e21-4175-a760-60619be198cf\" (UID: \"26623cb1-5e21-4175-a760-60619be198cf\") "
Oct 30 22:27:50.934796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.934769    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26623cb1-5e21-4175-a760-60619be198cf-openshift-service-ca" pod "26623cb1-5e21-4175-a760-60619be198cf" (UID: "26623cb1-5e21-4175-a760-60619be198cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:50.939106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.938857    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-metrics-certs" pod "26623cb1-5e21-4175-a760-60619be198cf" (UID: "26623cb1-5e21-4175-a760-60619be198cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:50.939349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.939325    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26623cb1-5e21-4175-a760-60619be198cf-kube-api-access-cn6d6" pod "26623cb1-5e21-4175-a760-60619be198cf" (UID: "26623cb1-5e21-4175-a760-60619be198cf"). InnerVolumeSpecName "kube-api-access-cn6d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:50.939422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:50.939403    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-maas-api-tls" pod "26623cb1-5e21-4175-a760-60619be198cf" (UID: "26623cb1-5e21-4175-a760-60619be198cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:51.053498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.052413    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26623cb1-5e21-4175-a760-60619be198cf-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:51.053498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.052480    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:51.053498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.052499    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26623cb1-5e21-4175-a760-60619be198cf-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:51.053498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.052514    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn6d6\" (UniqueName: \"kubernetes.io/projected/26623cb1-5e21-4175-a760-60619be198cf-kube-api-access-cn6d6\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:51.077451 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.077410    4282 generic.go:356] "Generic (PLEG): container finished" podID="26623cb1-5e21-4175-a760-60619be198cf" containerID="c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74" exitCode=0
Oct 30 22:27:51.077944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.077595    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5" event={"ID":"26623cb1-5e21-4175-a760-60619be198cf","Type":"ContainerDied","Data":"c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74"}
Oct 30 22:27:51.078163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.078145    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5" event={"ID":"26623cb1-5e21-4175-a760-60619be198cf","Type":"ContainerDied","Data":"8c63c64da2334119e019d94ab6013197a5553d3724b93b64d9f88bd1f10a7665"}
Oct 30 22:27:51.078292 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.078277    4282 scope.go:122] "RemoveContainer" containerID="c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74"
Oct 30 22:27:51.078519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.077690    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"
Oct 30 22:27:51.097721 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.097539    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" event={"ID":"de4440c1-907a-4752-9a69-197dfcb5ead4","Type":"ContainerStarted","Data":"0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf"}
Oct 30 22:27:51.097721 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.097604    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" event={"ID":"de4440c1-907a-4752-9a69-197dfcb5ead4","Type":"ContainerStarted","Data":"56fa78b05577860567b5962291ad8ed29eff88cc14d5e017917a70065f32c268"}
Oct 30 22:27:51.106007 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.105984    4282 scope.go:122] "RemoveContainer" containerID="c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74"
Oct 30 22:27:51.106314 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:27:51.106291    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74\": container with ID starting with c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74 not found: ID does not exist" containerID="c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74"
Oct 30 22:27:51.106410 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.106324    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74"} err="failed to get container status \"c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74\": rpc error: code = NotFound desc = could not find container \"c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74\": container with ID starting with c1ae026f51b60177a7049c736e2b4bd0712dd3054bc09c065e0a8ad0f1f86d74 not found: ID does not exist"
Oct 30 22:27:51.122272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.122205    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podStartSLOduration=2.122190969 podStartE2EDuration="2.122190969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:51.120406281 +0000 UTC m=+2889.778502504" watchObservedRunningTime="2026-10-30 22:27:51.122190969 +0000 UTC m=+2889.780287192"
Oct 30 22:27:51.138852 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.138818    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"]
Oct 30 22:27:51.148016 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.147988    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b2e72-86dd95768b-lz4c5"]
Oct 30 22:27:51.376542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.369498    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:51.376542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.371153    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:51.376542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.371951    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:27:51.520567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.520537    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26623cb1-5e21-4175-a760-60619be198cf" path="/var/lib/kubelet/pods/26623cb1-5e21-4175-a760-60619be198cf/volumes"
Oct 30 22:27:51.822319 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.822290    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:51.824187 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.824157    4282 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:27:51.824386 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.824368    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 22:27:51.913569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.913536    4282 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:27:51.913969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:51.913598    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:27:52.822302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:52.822260    4282 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:27:52.822516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:52.822309    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 22:27:52.914182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:52.914103    4282 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:27:52.914182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:52.914168    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:27:53.823544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:53.822846    4282 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:27:53.823544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:53.822903    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 22:27:53.914060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:53.914027    4282 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:27:53.914301 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:53.914276    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:27:54.823888 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:54.823845    4282 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:27:54.824087 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:54.823903    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 22:27:54.913947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:54.913912    4282 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:27:54.914166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:54.913971    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:27:55.822261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:55.822229    4282 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:27:55.822751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:55.822292    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 22:27:55.913511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:55.913459    4282 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Oct 30 22:27:55.913744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:55.913528    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Oct 30 22:27:56.822881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:56.822841    4282 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:27:56.823643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:56.823613    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 22:27:56.937592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:56.937545    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:56.939193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:56.939172    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:56.940182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:56.940160    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:27:57.029724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.029679    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"]
Oct 30 22:27:57.029956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.029907    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks" podUID="8ad28078-8c55-4d57-a2cf-01aa6339c0cd" containerName="sim" containerID="cri-o://65bd7ab43ff176f7968998346e1b6e546ed9a514447111f245e96cbd09b8a309" gracePeriod=30
Oct 30 22:27:57.240249 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.240217    4282 generic.go:356] "Generic (PLEG): container finished" podID="8ad28078-8c55-4d57-a2cf-01aa6339c0cd" containerID="65bd7ab43ff176f7968998346e1b6e546ed9a514447111f245e96cbd09b8a309" exitCode=0
Oct 30 22:27:57.240593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.240277    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks" event={"ID":"8ad28078-8c55-4d57-a2cf-01aa6339c0cd","Type":"ContainerDied","Data":"65bd7ab43ff176f7968998346e1b6e546ed9a514447111f245e96cbd09b8a309"}
Oct 30 22:27:57.738846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.738820    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:27:57.822548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.822515    4282 patch_prober.go:29] interesting pod/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:27:57.822734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.822571    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Oct 30 22:27:57.848284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.848253    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-tls\" (UniqueName: \"kubernetes.io/secret/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-tls\") pod \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\" (UID: \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\") "
Oct 30 22:27:57.848744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.848339    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-kube-api-access-nn2sr\" (UniqueName: \"kubernetes.io/projected/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-kube-api-access-nn2sr\") pod \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\" (UID: \"8ad28078-8c55-4d57-a2cf-01aa6339c0cd\") "
Oct 30 22:27:57.850743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.850685    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-tls" pod "8ad28078-8c55-4d57-a2cf-01aa6339c0cd" (UID: "8ad28078-8c55-4d57-a2cf-01aa6339c0cd"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:57.850868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.850784    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-kube-api-access-nn2sr" pod "8ad28078-8c55-4d57-a2cf-01aa6339c0cd" (UID: "8ad28078-8c55-4d57-a2cf-01aa6339c0cd"). InnerVolumeSpecName "kube-api-access-nn2sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:57.951769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.951730    4282 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:57.952122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:57.952062    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn2sr\" (UniqueName: \"kubernetes.io/projected/8ad28078-8c55-4d57-a2cf-01aa6339c0cd-kube-api-access-nn2sr\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.250905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.250830    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"
Oct 30 22:27:58.250905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.250861    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks" event={"ID":"8ad28078-8c55-4d57-a2cf-01aa6339c0cd","Type":"ContainerDied","Data":"36473e13bd7d0af9fb6df562233132e289d3436c26be291676dc8a81e347c8f3"}
Oct 30 22:27:58.251115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.250923    4282 scope.go:122] "RemoveContainer" containerID="65bd7ab43ff176f7968998346e1b6e546ed9a514447111f245e96cbd09b8a309"
Oct 30 22:27:58.282315 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.282282    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"]
Oct 30 22:27:58.287394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.287368    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-c10cdb/e2e-usage-sim-b15fa0-65f8d86c5c-965ks"]
Oct 30 22:27:58.826630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.826595    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"]
Oct 30 22:27:58.826928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.826848    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" containerID="cri-o://56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237" gracePeriod=30
Oct 30 22:27:58.828122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.828099    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:58.828579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.828540    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:58.829542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:58.829522    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:27:59.466695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:59.466663    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad28078-8c55-4d57-a2cf-01aa6339c0cd" path="/var/lib/kubelet/pods/8ad28078-8c55-4d57-a2cf-01aa6339c0cd/volumes"
Oct 30 22:27:59.662009 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:59.659858    4282 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:59.662009 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:27:59.659957    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:28:04.155316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.155283    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 22:28:04.252829 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.252791    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-podinfo\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.252869    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-kube-api-access-hjqd8\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-kube-api-access-hjqd8\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.252969    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-credential-socket\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253022    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-data\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253089    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-certs\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253202    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-envoy\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253249    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-credential-socket" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:04.253375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253297    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80220f4-6a6b-417e-ae8d-e373627a5425-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f80220f4-6a6b-417e-ae8d-e373627a5425-istiod-ca-cert\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253314    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-data" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:04.253375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253337    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-token\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-token\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253678 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253446    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-socket\") pod \"f80220f4-6a6b-417e-ae8d-e373627a5425\" (UID: \"f80220f4-6a6b-417e-ae8d-e373627a5425\") "
Oct 30 22:28:04.253678 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253505    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-certs" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:04.253993 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.253967    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80220f4-6a6b-417e-ae8d-e373627a5425-istiod-ca-cert" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:04.254431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.254402    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-socket" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:04.254641 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.254621    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.254724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.254648    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f80220f4-6a6b-417e-ae8d-e373627a5425-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.254724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.254664    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.254724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.254678    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.254724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.254692    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.255420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.255400    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-podinfo" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:28:04.255627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.255607    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-envoy" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:28:04.256000 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.255976    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-token" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:04.256099 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.256037    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-kube-api-access-hjqd8" pod "f80220f4-6a6b-417e-ae8d-e373627a5425" (UID: "f80220f4-6a6b-417e-ae8d-e373627a5425"). InnerVolumeSpecName "kube-api-access-hjqd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:04.354948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.354916    4282 generic.go:356] "Generic (PLEG): container finished" podID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerID="56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237" exitCode=0
Oct 30 22:28:04.355135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.354986    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"
Oct 30 22:28:04.355135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.354996    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" event={"ID":"f80220f4-6a6b-417e-ae8d-e373627a5425","Type":"ContainerDied","Data":"56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237"}
Oct 30 22:28:04.355135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.355057    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl" event={"ID":"f80220f4-6a6b-417e-ae8d-e373627a5425","Type":"ContainerDied","Data":"37a7e36689ccfca3be2b935b2c915ff45692545bdedf748df7c07f710afc0df0"}
Oct 30 22:28:04.355135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.355081    4282 scope.go:122] "RemoveContainer" containerID="56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237"
Oct 30 22:28:04.355968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.355943    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.356091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.355973    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjqd8\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-kube-api-access-hjqd8\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.356091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.355989    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.356091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.356004    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f80220f4-6a6b-417e-ae8d-e373627a5425-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.371848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.371828    4282 scope.go:122] "RemoveContainer" containerID="56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237"
Oct 30 22:28:04.372153 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:28:04.372131    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237\": container with ID starting with 56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237 not found: ID does not exist" containerID="56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237"
Oct 30 22:28:04.372206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.372163    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237"} err="failed to get container status \"56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237\": rpc error: code = NotFound desc = could not find container \"56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237\": container with ID starting with 56526deb63989f3c72c11e62266a2dcb81f14efd6122de96825cb753767d1237 not found: ID does not exist"
Oct 30 22:28:04.395432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.395408    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"]
Oct 30 22:28:04.402071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:04.402047    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-2h4wl"]
Oct 30 22:28:04.681693 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:28:04.681655    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:28:05.466101 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:05.466071    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" path="/var/lib/kubelet/pods/f80220f4-6a6b-417e-ae8d-e373627a5425/volumes"
Oct 30 22:28:34.650721 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:28:34.650682    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:28:40.246544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:40.246498    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:28:42.043166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:42.043113    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf" podUID="bc3156f5-7370-486a-89fb-65087b1f969e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.247:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:42.193616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:42.193566    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8" podUID="a002efa8-1326-4799-b1a9-c479bec25a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.248:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:42.285215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:42.285166    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:42.635494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:42.632534    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:28:43.045611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:43.045572    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf" podUID="bc3156f5-7370-486a-89fb-65087b1f969e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.247:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:43.046583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:43.046559    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:28:43.099388 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:43.099072    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:28:43.138667 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:43.137545    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:43.228703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:43.228666    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8" podUID="a002efa8-1326-4799-b1a9-c479bec25a4d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.248:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:28:43.265572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:43.265541    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:28:43.315221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:43.314994    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:28:59.170115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.170083    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"]
Oct 30 22:28:59.171123 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171098    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8ad28078-8c55-4d57-a2cf-01aa6339c0cd" containerName="sim"
Oct 30 22:28:59.171243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171123    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad28078-8c55-4d57-a2cf-01aa6339c0cd" containerName="sim"
Oct 30 22:28:59.171243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171178    4282 cpu_manager.go:414] "containerMap: removing container" podUID="26623cb1-5e21-4175-a760-60619be198cf" containerName="maas-api"
Oct 30 22:28:59.171243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171186    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26623cb1-5e21-4175-a760-60619be198cf" containerName="maas-api"
Oct 30 22:28:59.171243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171204    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy"
Oct 30 22:28:59.171243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171212    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy"
Oct 30 22:28:59.171546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171419    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad28078-8c55-4d57-a2cf-01aa6339c0cd" containerName="sim"
Oct 30 22:28:59.171546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171444    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="26623cb1-5e21-4175-a760-60619be198cf" containerName="maas-api"
Oct 30 22:28:59.171546 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.171457    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80220f4-6a6b-417e-ae8d-e373627a5425" containerName="istio-proxy"
Oct 30 22:28:59.177075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.177053    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.191452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.191429    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"]
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.355567    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.355672    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.355821    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.355859    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pplkw\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-kube-api-access-pplkw\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.355886    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.355936    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.356076    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.356177    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.360489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.356246    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458115    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458279    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458317    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pplkw\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-kube-api-access-pplkw\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458345    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458394    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458547    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458661    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458733    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458886 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458775    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458886 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458791    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.458886 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.458822    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.459106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.459023    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.459106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.459089    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.459618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.459299    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.461139 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.461112    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.461396 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.461371    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.469907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.469886    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.470061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.470043    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pplkw\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-kube-api-access-pplkw\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.493377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.493354    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:28:59.797320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.797288    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"]
Oct 30 22:28:59.798863 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:28:59.798836    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733b9d66_0f71_4d1a_9f57_8bd277cc7e2d.slice/crio-2b131d1e4e9ef0f1238f78df7d9be99a9e8b480283814d63f21cbdd12c11b70a WatchSource:0}: Error finding container 2b131d1e4e9ef0f1238f78df7d9be99a9e8b480283814d63f21cbdd12c11b70a: Status 404 returned error can't find the container with id 2b131d1e4e9ef0f1238f78df7d9be99a9e8b480283814d63f21cbdd12c11b70a
Oct 30 22:28:59.801769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.801730    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:28:59.801877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.801803    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:28:59.801877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:28:59.801834    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:29:00.037850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:00.037811    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" event={"ID":"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d","Type":"ContainerStarted","Data":"494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d"}
Oct 30 22:29:00.038065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:00.037875    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" event={"ID":"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d","Type":"ContainerStarted","Data":"2b131d1e4e9ef0f1238f78df7d9be99a9e8b480283814d63f21cbdd12c11b70a"}
Oct 30 22:29:00.061103 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:00.060381    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podStartSLOduration=1.060359902 podStartE2EDuration="1.060359902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:00.059027042 +0000 UTC m=+2958.717123265" watchObservedRunningTime="2026-10-30 22:29:00.060359902 +0000 UTC m=+2958.718456127"
Oct 30 22:29:00.493571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:00.493532    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:29:00.495088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:00.495063    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:29:00.495217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:00.495114    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 22:29:01.494116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:01.494081    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:29:01.494568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:01.494128    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 22:29:02.494121 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:02.494080    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:29:02.494541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:02.494143    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 22:29:03.494186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:03.494155    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:29:03.494617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:03.494206    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 22:29:04.497514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:04.494936    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:29:04.497514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:04.494994    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 22:29:04.689176 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:29:04.689139    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:29:05.494823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:05.494792    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:29:05.495035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:05.494847    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused"
Oct 30 22:29:06.499137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:06.499103    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:29:06.499756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:06.499412    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:29:06.500267 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:06.500246    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:29:12.427895 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.427858    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"]
Oct 30 22:29:12.434380 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.434345    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.439646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.439618    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-06ac2e-serving-cert\""
Oct 30 22:29:12.445067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.445038    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"]
Oct 30 22:29:12.561493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.560848    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66680916-61a4-451c-b6d8-42a57d23705d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.561493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.561011    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vjx\" (UniqueName: \"kubernetes.io/projected/66680916-61a4-451c-b6d8-42a57d23705d-kube-api-access-k2vjx\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.561493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.561168    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-metrics-certs\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.561493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.561332    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-maas-api-tls\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.663088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.663059    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66680916-61a4-451c-b6d8-42a57d23705d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.663277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.663160    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vjx\" (UniqueName: \"kubernetes.io/projected/66680916-61a4-451c-b6d8-42a57d23705d-kube-api-access-k2vjx\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.663277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.663199    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-metrics-certs\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.663370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.663288    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-maas-api-tls\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.663877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.663847    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66680916-61a4-451c-b6d8-42a57d23705d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.666001 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.665971    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-metrics-certs\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.666105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.666017    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-maas-api-tls\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.676919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.676897    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vjx\" (UniqueName: \"kubernetes.io/projected/66680916-61a4-451c-b6d8-42a57d23705d-kube-api-access-k2vjx\") pod \"maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:12.758462 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:12.758393    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:13.163091 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:13.163057    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"]
Oct 30 22:29:13.167841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:13.167819    4282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:29:13.183974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:13.183947    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb" event={"ID":"66680916-61a4-451c-b6d8-42a57d23705d","Type":"ContainerStarted","Data":"d46270278dbea86eb5e36e92381c1065b47baa18e13c62a1a40e626c4765e8d1"}
Oct 30 22:29:14.196110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:14.196074    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb" event={"ID":"66680916-61a4-451c-b6d8-42a57d23705d","Type":"ContainerStarted","Data":"931a8f20f2e69b3254f858bdfe49c73e016dc15e4ab2b5ec779bd8d8e3ad420d"}
Oct 30 22:29:14.196636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:14.196403    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:14.223172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:14.223122    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb" podStartSLOduration=1.7954152859999999 podStartE2EDuration="2.223108822s" totalImagesPullingTime="427.693536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:14.219706327 +0000 UTC m=+2972.877802550" watchObservedRunningTime="2026-10-30 22:29:14.223108822 +0000 UTC m=+2972.881205045"
Oct 30 22:29:20.210498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:20.210428    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:29:34.652459 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:29:34.652413    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:29:39.665711 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:29:39.665680    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:30:00.192942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.192846    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"]
Oct 30 22:30:00.199178 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.199153    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:00.233924 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.233885    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlwl\" (UniqueName: \"kubernetes.io/projected/81c56d6f-0193-4922-a60f-b0aa5a71910b-kube-api-access-2qlwl\") pod \"maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj\" (UID: \"81c56d6f-0193-4922-a60f-b0aa5a71910b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:00.234137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.233947    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c56d6f-0193-4922-a60f-b0aa5a71910b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj\" (UID: \"81c56d6f-0193-4922-a60f-b0aa5a71910b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:00.280990 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.280939    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\""
Oct 30 22:30:00.281314 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.281295    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"]
Oct 30 22:30:00.354968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.354916    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlwl\" (UniqueName: \"kubernetes.io/projected/81c56d6f-0193-4922-a60f-b0aa5a71910b-kube-api-access-2qlwl\") pod \"maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj\" (UID: \"81c56d6f-0193-4922-a60f-b0aa5a71910b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:00.394933 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.394888    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c56d6f-0193-4922-a60f-b0aa5a71910b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj\" (UID: \"81c56d6f-0193-4922-a60f-b0aa5a71910b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:00.396232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.396182    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c56d6f-0193-4922-a60f-b0aa5a71910b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj\" (UID: \"81c56d6f-0193-4922-a60f-b0aa5a71910b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:00.396409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.396226    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"]
Oct 30 22:30:00.396409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.396334    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"]
Oct 30 22:30:00.399526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.396794    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:00.410673 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.410637    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:00.432329 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.432289    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlwl\" (UniqueName: \"kubernetes.io/projected/81c56d6f-0193-4922-a60f-b0aa5a71910b-kube-api-access-2qlwl\") pod \"maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj\" (UID: \"81c56d6f-0193-4922-a60f-b0aa5a71910b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:00.444783 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.440675    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"]
Oct 30 22:30:00.455746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.455717    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"]
Oct 30 22:30:00.456046 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.456021    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:00.465576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.465542    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"]
Oct 30 22:30:00.473413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.473390    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"]
Oct 30 22:30:00.483953 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.483782    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"]
Oct 30 22:30:00.484138 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.483918    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:00.486379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.486351    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"]
Oct 30 22:30:00.493883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.493851    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"]
Oct 30 22:30:00.494051 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.494037    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:00.496717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.496687    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9d37c2-8e42-411e-8e26-85a048947be6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6\" (UID: \"be9d37c2-8e42-411e-8e26-85a048947be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:00.497147 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.497123    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffb1e-2508-4081-90e3-69febec2c84f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt\" (UID: \"c43ffb1e-2508-4081-90e3-69febec2c84f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:00.497393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.497373    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sc4h\" (UniqueName: \"kubernetes.io/projected/c43ffb1e-2508-4081-90e3-69febec2c84f-kube-api-access-2sc4h\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt\" (UID: \"c43ffb1e-2508-4081-90e3-69febec2c84f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:00.498018 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.497890    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdkr4\" (UniqueName: \"kubernetes.io/projected/be9d37c2-8e42-411e-8e26-85a048947be6-kube-api-access-rdkr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6\" (UID: \"be9d37c2-8e42-411e-8e26-85a048947be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:00.500387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.500354    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"]
Oct 30 22:30:00.500814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.500676    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:00.507651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.507625    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"]
Oct 30 22:30:00.507772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.507738    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:00.512759 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.512740    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"]
Oct 30 22:30:00.520875 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.520851    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:00.522235 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.521545    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"]
Oct 30 22:30:00.528264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.528245    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"]
Oct 30 22:30:00.528381 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.528367    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:00.534740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.533823    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"]
Oct 30 22:30:00.534740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.533925    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:00.537539 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.537512    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"]
Oct 30 22:30:00.542784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.542761    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"]
Oct 30 22:30:00.547317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.547289    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"]
Oct 30 22:30:00.602084 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602053    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxncj\" (UniqueName: \"kubernetes.io/projected/14311e13-e401-442b-814e-4aa4e751e23c-kube-api-access-cxncj\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d\" (UID: \"14311e13-e401-442b-814e-4aa4e751e23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:00.602269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602139    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwx5\" (UniqueName: \"kubernetes.io/projected/d2041356-82e2-4aaf-bf87-0ac90df55ba1-kube-api-access-wdwx5\") pod \"maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb\" (UID: \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:00.602269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602191    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkr4\" (UniqueName: \"kubernetes.io/projected/be9d37c2-8e42-411e-8e26-85a048947be6-kube-api-access-rdkr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6\" (UID: \"be9d37c2-8e42-411e-8e26-85a048947be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:00.602269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602215    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdjl\" (UniqueName: \"kubernetes.io/projected/19855a1c-4f0c-478c-acf4-8eb39097fc92-kube-api-access-6vdjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt\" (UID: \"19855a1c-4f0c-478c-acf4-8eb39097fc92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:00.602269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602251    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28591d5d-928e-407f-b292-81ce32dc3393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj\" (UID: \"28591d5d-928e-407f-b292-81ce32dc3393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602300    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9d37c2-8e42-411e-8e26-85a048947be6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6\" (UID: \"be9d37c2-8e42-411e-8e26-85a048947be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602352    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14311e13-e401-442b-814e-4aa4e751e23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d\" (UID: \"14311e13-e401-442b-814e-4aa4e751e23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602386    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8410d5b0-f783-412b-8075-f83b710df94e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72\" (UID: \"8410d5b0-f783-412b-8075-f83b710df94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602439    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjbq\" (UniqueName: \"kubernetes.io/projected/0c2ce866-ccbc-4279-915e-cb75f1338701-kube-api-access-zmjbq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck\" (UID: \"0c2ce866-ccbc-4279-915e-cb75f1338701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602563    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7\" (UID: \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602629    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2041356-82e2-4aaf-bf87-0ac90df55ba1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb\" (UID: \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602651    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ce866-ccbc-4279-915e-cb75f1338701-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck\" (UID: \"0c2ce866-ccbc-4279-915e-cb75f1338701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602682    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7549d\" (UniqueName: \"kubernetes.io/projected/28591d5d-928e-407f-b292-81ce32dc3393-kube-api-access-7549d\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj\" (UID: \"28591d5d-928e-407f-b292-81ce32dc3393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602726    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk62d\" (UniqueName: \"kubernetes.io/projected/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-kube-api-access-hk62d\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7\" (UID: \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602754    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffb1e-2508-4081-90e3-69febec2c84f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt\" (UID: \"c43ffb1e-2508-4081-90e3-69febec2c84f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602809    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsw4b\" (UniqueName: \"kubernetes.io/projected/8410d5b0-f783-412b-8075-f83b710df94e-kube-api-access-xsw4b\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72\" (UID: \"8410d5b0-f783-412b-8075-f83b710df94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602846    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sc4h\" (UniqueName: \"kubernetes.io/projected/c43ffb1e-2508-4081-90e3-69febec2c84f-kube-api-access-2sc4h\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt\" (UID: \"c43ffb1e-2508-4081-90e3-69febec2c84f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:00.603119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.602877    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19855a1c-4f0c-478c-acf4-8eb39097fc92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt\" (UID: \"19855a1c-4f0c-478c-acf4-8eb39097fc92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:00.604151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.604111    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9d37c2-8e42-411e-8e26-85a048947be6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6\" (UID: \"be9d37c2-8e42-411e-8e26-85a048947be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:00.604908 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.604885    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffb1e-2508-4081-90e3-69febec2c84f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt\" (UID: \"c43ffb1e-2508-4081-90e3-69febec2c84f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:00.634526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.619417    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sc4h\" (UniqueName: \"kubernetes.io/projected/c43ffb1e-2508-4081-90e3-69febec2c84f-kube-api-access-2sc4h\") pod \"maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt\" (UID: \"c43ffb1e-2508-4081-90e3-69febec2c84f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:00.634526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.631814    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkr4\" (UniqueName: \"kubernetes.io/projected/be9d37c2-8e42-411e-8e26-85a048947be6-kube-api-access-rdkr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6\" (UID: \"be9d37c2-8e42-411e-8e26-85a048947be6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:00.704273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704165    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2041356-82e2-4aaf-bf87-0ac90df55ba1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb\" (UID: \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:00.704273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704211    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ce866-ccbc-4279-915e-cb75f1338701-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck\" (UID: \"0c2ce866-ccbc-4279-915e-cb75f1338701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:00.704273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704236    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7549d\" (UniqueName: \"kubernetes.io/projected/28591d5d-928e-407f-b292-81ce32dc3393-kube-api-access-7549d\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj\" (UID: \"28591d5d-928e-407f-b292-81ce32dc3393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:00.704768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704277    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk62d\" (UniqueName: \"kubernetes.io/projected/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-kube-api-access-hk62d\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7\" (UID: \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:00.704768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704322    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsw4b\" (UniqueName: \"kubernetes.io/projected/8410d5b0-f783-412b-8075-f83b710df94e-kube-api-access-xsw4b\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72\" (UID: \"8410d5b0-f783-412b-8075-f83b710df94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:00.704768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704402    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19855a1c-4f0c-478c-acf4-8eb39097fc92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt\" (UID: \"19855a1c-4f0c-478c-acf4-8eb39097fc92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:00.704768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704497    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxncj\" (UniqueName: \"kubernetes.io/projected/14311e13-e401-442b-814e-4aa4e751e23c-kube-api-access-cxncj\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d\" (UID: \"14311e13-e401-442b-814e-4aa4e751e23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:00.704768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704583    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwx5\" (UniqueName: \"kubernetes.io/projected/d2041356-82e2-4aaf-bf87-0ac90df55ba1-kube-api-access-wdwx5\") pod \"maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb\" (UID: \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:00.704768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704673    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdjl\" (UniqueName: \"kubernetes.io/projected/19855a1c-4f0c-478c-acf4-8eb39097fc92-kube-api-access-6vdjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt\" (UID: \"19855a1c-4f0c-478c-acf4-8eb39097fc92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:00.704768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704736    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28591d5d-928e-407f-b292-81ce32dc3393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj\" (UID: \"28591d5d-928e-407f-b292-81ce32dc3393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:00.705182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704862    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14311e13-e401-442b-814e-4aa4e751e23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d\" (UID: \"14311e13-e401-442b-814e-4aa4e751e23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:00.705182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704926    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8410d5b0-f783-412b-8075-f83b710df94e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72\" (UID: \"8410d5b0-f783-412b-8075-f83b710df94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:00.705182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.704997    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjbq\" (UniqueName: \"kubernetes.io/projected/0c2ce866-ccbc-4279-915e-cb75f1338701-kube-api-access-zmjbq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck\" (UID: \"0c2ce866-ccbc-4279-915e-cb75f1338701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:00.705182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.705076    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ce866-ccbc-4279-915e-cb75f1338701-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck\" (UID: \"0c2ce866-ccbc-4279-915e-cb75f1338701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:00.705182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.705096    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7\" (UID: \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:00.705182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.705076    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2041356-82e2-4aaf-bf87-0ac90df55ba1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb\" (UID: \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:00.705702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.705402    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19855a1c-4f0c-478c-acf4-8eb39097fc92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt\" (UID: \"19855a1c-4f0c-478c-acf4-8eb39097fc92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:00.705915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.705894    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8410d5b0-f783-412b-8075-f83b710df94e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72\" (UID: \"8410d5b0-f783-412b-8075-f83b710df94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:00.705995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.705897    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14311e13-e401-442b-814e-4aa4e751e23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d\" (UID: \"14311e13-e401-442b-814e-4aa4e751e23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:00.705995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.705916    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7\" (UID: \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:00.705995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.705953    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28591d5d-928e-407f-b292-81ce32dc3393-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj\" (UID: \"28591d5d-928e-407f-b292-81ce32dc3393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:00.717566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.717533    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwx5\" (UniqueName: \"kubernetes.io/projected/d2041356-82e2-4aaf-bf87-0ac90df55ba1-kube-api-access-wdwx5\") pod \"maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb\" (UID: \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:00.719615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.719301    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk62d\" (UniqueName: \"kubernetes.io/projected/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-kube-api-access-hk62d\") pod \"maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7\" (UID: \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:00.719615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.719301    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsw4b\" (UniqueName: \"kubernetes.io/projected/8410d5b0-f783-412b-8075-f83b710df94e-kube-api-access-xsw4b\") pod \"maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72\" (UID: \"8410d5b0-f783-412b-8075-f83b710df94e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:00.719615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.719567    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjbq\" (UniqueName: \"kubernetes.io/projected/0c2ce866-ccbc-4279-915e-cb75f1338701-kube-api-access-zmjbq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck\" (UID: \"0c2ce866-ccbc-4279-915e-cb75f1338701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:00.719883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.719770    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxncj\" (UniqueName: \"kubernetes.io/projected/14311e13-e401-442b-814e-4aa4e751e23c-kube-api-access-cxncj\") pod \"maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d\" (UID: \"14311e13-e401-442b-814e-4aa4e751e23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:00.721021 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.720995    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdjl\" (UniqueName: \"kubernetes.io/projected/19855a1c-4f0c-478c-acf4-8eb39097fc92-kube-api-access-6vdjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt\" (UID: \"19855a1c-4f0c-478c-acf4-8eb39097fc92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:00.721682 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.721659    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7549d\" (UniqueName: \"kubernetes.io/projected/28591d5d-928e-407f-b292-81ce32dc3393-kube-api-access-7549d\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj\" (UID: \"28591d5d-928e-407f-b292-81ce32dc3393\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:00.772648 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.772613    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:00.783020 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.782996    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:00.795071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.795042    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:00.813987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.813955    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:00.828130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.825444    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:00.834899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.834854    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"]
Oct 30 22:30:00.839620 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:30:00.839587    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c56d6f_0193_4922_a60f_b0aa5a71910b.slice/crio-5c1475dbe639f9fd163146794d481846cabc2c90cd9ee152dc5e0d67dda36a33 WatchSource:0}: Error finding container 5c1475dbe639f9fd163146794d481846cabc2c90cd9ee152dc5e0d67dda36a33: Status 404 returned error can't find the container with id 5c1475dbe639f9fd163146794d481846cabc2c90cd9ee152dc5e0d67dda36a33
Oct 30 22:30:00.840018 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.839987    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:00.891658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.888739    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:00.891658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.889663    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:00.916014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:00.915986    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:01.330298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.330262    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"]
Oct 30 22:30:01.336978 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:30:01.335774    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc43ffb1e_2508_4081_90e3_69febec2c84f.slice/crio-fe96e488f464c180311791ae3e6661799823bdcc85ab004778f7fdcf64396b0d WatchSource:0}: Error finding container fe96e488f464c180311791ae3e6661799823bdcc85ab004778f7fdcf64396b0d: Status 404 returned error can't find the container with id fe96e488f464c180311791ae3e6661799823bdcc85ab004778f7fdcf64396b0d
Oct 30 22:30:01.349801 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.349698    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"]
Oct 30 22:30:01.377881 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:30:01.374593    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9d37c2_8e42_411e_8e26_85a048947be6.slice/crio-e7a056e11c69a159442d06c08689dd752b0fc82f806f82c30b9939d7f1d17019 WatchSource:0}: Error finding container e7a056e11c69a159442d06c08689dd752b0fc82f806f82c30b9939d7f1d17019: Status 404 returned error can't find the container with id e7a056e11c69a159442d06c08689dd752b0fc82f806f82c30b9939d7f1d17019
Oct 30 22:30:01.414075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.414032    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"]
Oct 30 22:30:01.430999 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:30:01.430678    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122c7ccb_de31_4bba_9b54_bbc4ad6a32c8.slice/crio-dcf94176f4cc9a8426315a77bb6ea45717dd4e8715e23d81f822651c77360978 WatchSource:0}: Error finding container dcf94176f4cc9a8426315a77bb6ea45717dd4e8715e23d81f822651c77360978: Status 404 returned error can't find the container with id dcf94176f4cc9a8426315a77bb6ea45717dd4e8715e23d81f822651c77360978
Oct 30 22:30:01.430999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.430881    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"]
Oct 30 22:30:01.507594 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.507565    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"]
Oct 30 22:30:01.514385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.514354    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"]
Oct 30 22:30:01.536389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.536364    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"]
Oct 30 22:30:01.547873 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.545331    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"]
Oct 30 22:30:01.636274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.636244    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"]
Oct 30 22:30:01.637149 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:30:01.637126    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28591d5d_928e_407f_b292_81ce32dc3393.slice/crio-1c175747b20e453835ceb8504ae3c86eaac0e48a08486ff4e3dd46c07f317967 WatchSource:0}: Error finding container 1c175747b20e453835ceb8504ae3c86eaac0e48a08486ff4e3dd46c07f317967: Status 404 returned error can't find the container with id 1c175747b20e453835ceb8504ae3c86eaac0e48a08486ff4e3dd46c07f317967
Oct 30 22:30:01.756186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.756148    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" event={"ID":"d2041356-82e2-4aaf-bf87-0ac90df55ba1","Type":"ContainerStarted","Data":"775a1aacc9eafc8bd8447c12459ceb9c1a89b48222a4e3fc8015855f1f50add4"}
Oct 30 22:30:01.757872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.757838    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" event={"ID":"28591d5d-928e-407f-b292-81ce32dc3393","Type":"ContainerStarted","Data":"1c175747b20e453835ceb8504ae3c86eaac0e48a08486ff4e3dd46c07f317967"}
Oct 30 22:30:01.759317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.759285    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" event={"ID":"14311e13-e401-442b-814e-4aa4e751e23c","Type":"ContainerStarted","Data":"5b7e7055c4db0d507886a1f08846a19824eee208e29b5cb76fef57dd06e5e253"}
Oct 30 22:30:01.760661 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.760637    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" event={"ID":"c43ffb1e-2508-4081-90e3-69febec2c84f","Type":"ContainerStarted","Data":"fe96e488f464c180311791ae3e6661799823bdcc85ab004778f7fdcf64396b0d"}
Oct 30 22:30:01.762232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.762210    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" event={"ID":"8410d5b0-f783-412b-8075-f83b710df94e","Type":"ContainerStarted","Data":"87617039754a24ddd29683b299c60b25436db7f730a1b69e5b2eaf592fb63634"}
Oct 30 22:30:01.763896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.763870    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" event={"ID":"19855a1c-4f0c-478c-acf4-8eb39097fc92","Type":"ContainerStarted","Data":"216a56b2bfaef88f633b9ae9652c936d6021395bb9263210ed3f6f211a990351"}
Oct 30 22:30:01.765955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.765932    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" event={"ID":"81c56d6f-0193-4922-a60f-b0aa5a71910b","Type":"ContainerStarted","Data":"7630fd96d8754673effcaf67bb30d8182e75cf93ded159ebe93758ccda0582d7"}
Oct 30 22:30:01.766057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.765976    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" event={"ID":"81c56d6f-0193-4922-a60f-b0aa5a71910b","Type":"ContainerStarted","Data":"5c1475dbe639f9fd163146794d481846cabc2c90cd9ee152dc5e0d67dda36a33"}
Oct 30 22:30:01.767665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.767627    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" event={"ID":"be9d37c2-8e42-411e-8e26-85a048947be6","Type":"ContainerStarted","Data":"e7a056e11c69a159442d06c08689dd752b0fc82f806f82c30b9939d7f1d17019"}
Oct 30 22:30:01.769031 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.769004    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" event={"ID":"0c2ce866-ccbc-4279-915e-cb75f1338701","Type":"ContainerStarted","Data":"7ba770097f2b21e3b943381fb9fe2cb0cd8d8a258890177681892f984be925ed"}
Oct 30 22:30:01.770326 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.770303    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" event={"ID":"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8","Type":"ContainerStarted","Data":"dcf94176f4cc9a8426315a77bb6ea45717dd4e8715e23d81f822651c77360978"}
Oct 30 22:30:01.786200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:01.786121    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" podStartSLOduration=1.786109406 podStartE2EDuration="1.786109406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.781789952 +0000 UTC m=+3020.439886175" watchObservedRunningTime="2026-10-30 22:30:01.786109406 +0000 UTC m=+3020.444205628"
Oct 30 22:30:02.781050 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.781013    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" event={"ID":"d2041356-82e2-4aaf-bf87-0ac90df55ba1","Type":"ContainerStarted","Data":"8bf62d9b65f582e3d714983e28fd11ca1a9888af32e87c39c8445d9ede1441d0"}
Oct 30 22:30:02.782722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.782694    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" event={"ID":"28591d5d-928e-407f-b292-81ce32dc3393","Type":"ContainerStarted","Data":"2df0d7a491e00f8f3b495a699e7978706bba2f7546ab42bf31b67eb64e3436bc"}
Oct 30 22:30:02.784411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.784387    4282 generic.go:356] "Generic (PLEG): container finished" podID="14311e13-e401-442b-814e-4aa4e751e23c" containerID="e2aad1030c008d072a4ce5f5125b7e14cacefc749e3383c8e29598d3ded83c8f" exitCode=60
Oct 30 22:30:02.784567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.784457    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" event={"ID":"14311e13-e401-442b-814e-4aa4e751e23c","Type":"ContainerDied","Data":"e2aad1030c008d072a4ce5f5125b7e14cacefc749e3383c8e29598d3ded83c8f"}
Oct 30 22:30:02.784770 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.784751    4282 scope.go:122] "RemoveContainer" containerID="e2aad1030c008d072a4ce5f5125b7e14cacefc749e3383c8e29598d3ded83c8f"
Oct 30 22:30:02.786106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.786082    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" event={"ID":"c43ffb1e-2508-4081-90e3-69febec2c84f","Type":"ContainerStarted","Data":"5ead63034c6940b70afe46a35bbd00723c7d5cd9606666080a7b2cce5687f941"}
Oct 30 22:30:02.792432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.792404    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" event={"ID":"8410d5b0-f783-412b-8075-f83b710df94e","Type":"ContainerStarted","Data":"bee67ff5f895e3ad50fe38ccd69eee2353bf33287235245c7f6c2b1a440e5307"}
Oct 30 22:30:02.794755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.794729    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" event={"ID":"19855a1c-4f0c-478c-acf4-8eb39097fc92","Type":"ContainerStarted","Data":"1c6f81ad1014b3e39e0408f42b2ff08bc743831ebc74ba5b09c7da9b893551ed"}
Oct 30 22:30:02.796713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.796689    4282 generic.go:356] "Generic (PLEG): container finished" podID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerID="7630fd96d8754673effcaf67bb30d8182e75cf93ded159ebe93758ccda0582d7" exitCode=60
Oct 30 22:30:02.796825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.796746    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" event={"ID":"81c56d6f-0193-4922-a60f-b0aa5a71910b","Type":"ContainerDied","Data":"7630fd96d8754673effcaf67bb30d8182e75cf93ded159ebe93758ccda0582d7"}
Oct 30 22:30:02.797034 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.796964    4282 scope.go:122] "RemoveContainer" containerID="7630fd96d8754673effcaf67bb30d8182e75cf93ded159ebe93758ccda0582d7"
Oct 30 22:30:02.797899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.797852    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" podStartSLOduration=2.797836879 podStartE2EDuration="2.797836879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.79703401 +0000 UTC m=+3021.455130235" watchObservedRunningTime="2026-10-30 22:30:02.797836879 +0000 UTC m=+3021.455933102"
Oct 30 22:30:02.801425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.801404    4282 generic.go:356] "Generic (PLEG): container finished" podID="be9d37c2-8e42-411e-8e26-85a048947be6" containerID="a8b6b5f9a347e8dcd66ced28d5af13a414ecee8c3abee175313b8b4628b14c03" exitCode=60
Oct 30 22:30:02.801563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.801498    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" event={"ID":"be9d37c2-8e42-411e-8e26-85a048947be6","Type":"ContainerDied","Data":"a8b6b5f9a347e8dcd66ced28d5af13a414ecee8c3abee175313b8b4628b14c03"}
Oct 30 22:30:02.801738 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.801719    4282 scope.go:122] "RemoveContainer" containerID="a8b6b5f9a347e8dcd66ced28d5af13a414ecee8c3abee175313b8b4628b14c03"
Oct 30 22:30:02.803451 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.803428    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" event={"ID":"0c2ce866-ccbc-4279-915e-cb75f1338701","Type":"ContainerStarted","Data":"ca040500276cf0351cc371bd67e13a28de4feb6ca2e2738dda07315da773b0b1"}
Oct 30 22:30:02.805272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.805249    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" event={"ID":"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8","Type":"ContainerStarted","Data":"1c6b3ba0b45feef7c6e113027c9f1cfab581e70f4fe8286deaacfef9b8a62613"}
Oct 30 22:30:02.880344 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.861412    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" podStartSLOduration=2.861391666 podStartE2EDuration="2.861391666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.812740819 +0000 UTC m=+3021.470837044" watchObservedRunningTime="2026-10-30 22:30:02.861391666 +0000 UTC m=+3021.519487890"
Oct 30 22:30:02.910751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.910708    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" podStartSLOduration=2.910691957 podStartE2EDuration="2.910691957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.873548673 +0000 UTC m=+3021.531645165" watchObservedRunningTime="2026-10-30 22:30:02.910691957 +0000 UTC m=+3021.568788181"
Oct 30 22:30:02.928616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.928573    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" podStartSLOduration=2.928558571 podStartE2EDuration="2.928558571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.920043711 +0000 UTC m=+3021.578139935" watchObservedRunningTime="2026-10-30 22:30:02.928558571 +0000 UTC m=+3021.586654793"
Oct 30 22:30:02.941354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.941315    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" podStartSLOduration=2.9412989510000003 podStartE2EDuration="2.941298951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.937300537 +0000 UTC m=+3021.595396762" watchObservedRunningTime="2026-10-30 22:30:02.941298951 +0000 UTC m=+3021.599395205"
Oct 30 22:30:02.977327 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.977267    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" podStartSLOduration=2.977253503 podStartE2EDuration="2.977253503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.976113981 +0000 UTC m=+3021.634210204" watchObservedRunningTime="2026-10-30 22:30:02.977253503 +0000 UTC m=+3021.635349727"
Oct 30 22:30:02.995823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:02.995759    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" podStartSLOduration=2.995742387 podStartE2EDuration="2.995742387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:02.992062659 +0000 UTC m=+3021.650158886" watchObservedRunningTime="2026-10-30 22:30:02.995742387 +0000 UTC m=+3021.653838610"
Oct 30 22:30:03.816651 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.816615    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" event={"ID":"81c56d6f-0193-4922-a60f-b0aa5a71910b","Type":"ContainerStarted","Data":"073f03a885cbc288bfb004afad53a8e7166835ef38349a9b6211275b2c3ca905"}
Oct 30 22:30:03.818788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.818758    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" event={"ID":"be9d37c2-8e42-411e-8e26-85a048947be6","Type":"ContainerStarted","Data":"6ffc2ad4ea260ed133f4aa576a1ba02da238aed415f709a94d37fa6d9e6880c7"}
Oct 30 22:30:03.820519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.820492    4282 generic.go:356] "Generic (PLEG): container finished" podID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerID="ca040500276cf0351cc371bd67e13a28de4feb6ca2e2738dda07315da773b0b1" exitCode=60
Oct 30 22:30:03.820654 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.820630    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" event={"ID":"0c2ce866-ccbc-4279-915e-cb75f1338701","Type":"ContainerDied","Data":"ca040500276cf0351cc371bd67e13a28de4feb6ca2e2738dda07315da773b0b1"}
Oct 30 22:30:03.820921 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.820903    4282 scope.go:122] "RemoveContainer" containerID="ca040500276cf0351cc371bd67e13a28de4feb6ca2e2738dda07315da773b0b1"
Oct 30 22:30:03.822506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.822440    4282 generic.go:356] "Generic (PLEG): container finished" podID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerID="1c6b3ba0b45feef7c6e113027c9f1cfab581e70f4fe8286deaacfef9b8a62613" exitCode=60
Oct 30 22:30:03.822616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.822530    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" event={"ID":"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8","Type":"ContainerDied","Data":"1c6b3ba0b45feef7c6e113027c9f1cfab581e70f4fe8286deaacfef9b8a62613"}
Oct 30 22:30:03.822850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.822835    4282 scope.go:122] "RemoveContainer" containerID="1c6b3ba0b45feef7c6e113027c9f1cfab581e70f4fe8286deaacfef9b8a62613"
Oct 30 22:30:03.824689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.824671    4282 generic.go:356] "Generic (PLEG): container finished" podID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerID="8bf62d9b65f582e3d714983e28fd11ca1a9888af32e87c39c8445d9ede1441d0" exitCode=60
Oct 30 22:30:03.824814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.824774    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" event={"ID":"d2041356-82e2-4aaf-bf87-0ac90df55ba1","Type":"ContainerDied","Data":"8bf62d9b65f582e3d714983e28fd11ca1a9888af32e87c39c8445d9ede1441d0"}
Oct 30 22:30:03.825023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.825005    4282 scope.go:122] "RemoveContainer" containerID="8bf62d9b65f582e3d714983e28fd11ca1a9888af32e87c39c8445d9ede1441d0"
Oct 30 22:30:03.826830 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.826808    4282 generic.go:356] "Generic (PLEG): container finished" podID="28591d5d-928e-407f-b292-81ce32dc3393" containerID="2df0d7a491e00f8f3b495a699e7978706bba2f7546ab42bf31b67eb64e3436bc" exitCode=60
Oct 30 22:30:03.826897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.826885    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" event={"ID":"28591d5d-928e-407f-b292-81ce32dc3393","Type":"ContainerDied","Data":"2df0d7a491e00f8f3b495a699e7978706bba2f7546ab42bf31b67eb64e3436bc"}
Oct 30 22:30:03.827157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.827133    4282 scope.go:122] "RemoveContainer" containerID="2df0d7a491e00f8f3b495a699e7978706bba2f7546ab42bf31b67eb64e3436bc"
Oct 30 22:30:03.829103 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.829082    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" event={"ID":"14311e13-e401-442b-814e-4aa4e751e23c","Type":"ContainerStarted","Data":"efd63847f335fd001f295b6e7f9a7297eac339189e6dfb38c3432389b956aeea"}
Oct 30 22:30:03.830856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.830835    4282 generic.go:356] "Generic (PLEG): container finished" podID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerID="5ead63034c6940b70afe46a35bbd00723c7d5cd9606666080a7b2cce5687f941" exitCode=60
Oct 30 22:30:03.830973 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.830919    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" event={"ID":"c43ffb1e-2508-4081-90e3-69febec2c84f","Type":"ContainerDied","Data":"5ead63034c6940b70afe46a35bbd00723c7d5cd9606666080a7b2cce5687f941"}
Oct 30 22:30:03.831160 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.831142    4282 scope.go:122] "RemoveContainer" containerID="5ead63034c6940b70afe46a35bbd00723c7d5cd9606666080a7b2cce5687f941"
Oct 30 22:30:03.832820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.832800    4282 generic.go:356] "Generic (PLEG): container finished" podID="8410d5b0-f783-412b-8075-f83b710df94e" containerID="bee67ff5f895e3ad50fe38ccd69eee2353bf33287235245c7f6c2b1a440e5307" exitCode=60
Oct 30 22:30:03.832936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.832860    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" event={"ID":"8410d5b0-f783-412b-8075-f83b710df94e","Type":"ContainerDied","Data":"bee67ff5f895e3ad50fe38ccd69eee2353bf33287235245c7f6c2b1a440e5307"}
Oct 30 22:30:03.833124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.833108    4282 scope.go:122] "RemoveContainer" containerID="bee67ff5f895e3ad50fe38ccd69eee2353bf33287235245c7f6c2b1a440e5307"
Oct 30 22:30:03.837785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.836217    4282 generic.go:356] "Generic (PLEG): container finished" podID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerID="1c6f81ad1014b3e39e0408f42b2ff08bc743831ebc74ba5b09c7da9b893551ed" exitCode=60
Oct 30 22:30:03.837785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.836255    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" event={"ID":"19855a1c-4f0c-478c-acf4-8eb39097fc92","Type":"ContainerDied","Data":"1c6f81ad1014b3e39e0408f42b2ff08bc743831ebc74ba5b09c7da9b893551ed"}
Oct 30 22:30:03.837785 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.836664    4282 scope.go:122] "RemoveContainer" containerID="1c6f81ad1014b3e39e0408f42b2ff08bc743831ebc74ba5b09c7da9b893551ed"
Oct 30 22:30:03.970163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.970110    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" podStartSLOduration=3.970097128 podStartE2EDuration="3.970097128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.966820942 +0000 UTC m=+3022.624917165" watchObservedRunningTime="2026-10-30 22:30:03.970097128 +0000 UTC m=+3022.628193350"
Oct 30 22:30:03.989175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:03.989128    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" podStartSLOduration=3.989114631 podStartE2EDuration="3.989114631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.984634944 +0000 UTC m=+3022.642731166" watchObservedRunningTime="2026-10-30 22:30:03.989114631 +0000 UTC m=+3022.647210854"
Oct 30 22:30:04.651821 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:04.651728    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:30:04.847200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.847167    4282 generic.go:356] "Generic (PLEG): container finished" podID="be9d37c2-8e42-411e-8e26-85a048947be6" containerID="6ffc2ad4ea260ed133f4aa576a1ba02da238aed415f709a94d37fa6d9e6880c7" exitCode=60
Oct 30 22:30:04.847668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.847248    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" event={"ID":"be9d37c2-8e42-411e-8e26-85a048947be6","Type":"ContainerDied","Data":"6ffc2ad4ea260ed133f4aa576a1ba02da238aed415f709a94d37fa6d9e6880c7"}
Oct 30 22:30:04.847668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.847309    4282 scope.go:122] "RemoveContainer" containerID="a8b6b5f9a347e8dcd66ced28d5af13a414ecee8c3abee175313b8b4628b14c03"
Oct 30 22:30:04.847668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.847621    4282 scope.go:122] "RemoveContainer" containerID="6ffc2ad4ea260ed133f4aa576a1ba02da238aed415f709a94d37fa6d9e6880c7"
Oct 30 22:30:04.848012 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:04.847984    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6_odh-ai-gateway-infra(be9d37c2-8e42-411e-8e26-85a048947be6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" podUID="be9d37c2-8e42-411e-8e26-85a048947be6"
Oct 30 22:30:04.849696 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.849671    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" event={"ID":"0c2ce866-ccbc-4279-915e-cb75f1338701","Type":"ContainerStarted","Data":"29bbce338fe576f78835435c82c0978b53433be7de560f1d368d5d81a262f11c"}
Oct 30 22:30:04.851898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.851855    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" event={"ID":"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8","Type":"ContainerStarted","Data":"dae5a9d71ee354ab296d238b3f15d9fe49d708b85ca39828e671d94d77408060"}
Oct 30 22:30:04.854686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.854661    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" event={"ID":"d2041356-82e2-4aaf-bf87-0ac90df55ba1","Type":"ContainerStarted","Data":"9e817ad5242dde8a2aaa03f6520fd84543c40d6b11ba5ad59f8d8bcf7d0b1e2c"}
Oct 30 22:30:04.857963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.857927    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" event={"ID":"28591d5d-928e-407f-b292-81ce32dc3393","Type":"ContainerStarted","Data":"238290933dd937fd967b31bfb1782570e3cf8c22f6fdc61fdff71b6d80c5e567"}
Oct 30 22:30:04.860344 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.860323    4282 generic.go:356] "Generic (PLEG): container finished" podID="14311e13-e401-442b-814e-4aa4e751e23c" containerID="efd63847f335fd001f295b6e7f9a7297eac339189e6dfb38c3432389b956aeea" exitCode=60
Oct 30 22:30:04.860428 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.860387    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" event={"ID":"14311e13-e401-442b-814e-4aa4e751e23c","Type":"ContainerDied","Data":"efd63847f335fd001f295b6e7f9a7297eac339189e6dfb38c3432389b956aeea"}
Oct 30 22:30:04.860680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.860662    4282 scope.go:122] "RemoveContainer" containerID="efd63847f335fd001f295b6e7f9a7297eac339189e6dfb38c3432389b956aeea"
Oct 30 22:30:04.860996 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:04.860972    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d_odh-ai-gateway-infra(14311e13-e401-442b-814e-4aa4e751e23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" podUID="14311e13-e401-442b-814e-4aa4e751e23c"
Oct 30 22:30:04.862277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.862256    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" event={"ID":"c43ffb1e-2508-4081-90e3-69febec2c84f","Type":"ContainerStarted","Data":"54aa3ac6b4ce3e3893039ac435a6d2e1a22cc2803c9e767a315eeb7c08fc5955"}
Oct 30 22:30:04.864353 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.864334    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" event={"ID":"8410d5b0-f783-412b-8075-f83b710df94e","Type":"ContainerStarted","Data":"948194d042d784bed61ca9864b1a8b376b8f697b05f61ec9030d52d02556dc18"}
Oct 30 22:30:04.866404 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.866384    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" event={"ID":"19855a1c-4f0c-478c-acf4-8eb39097fc92","Type":"ContainerStarted","Data":"a37e046269150ce2a182d9a86788d181e73d635cad9c54d77e771317eac49f18"}
Oct 30 22:30:04.866802 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.866786    4282 scope.go:122] "RemoveContainer" containerID="e2aad1030c008d072a4ce5f5125b7e14cacefc749e3383c8e29598d3ded83c8f"
Oct 30 22:30:04.869112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.869091    4282 generic.go:356] "Generic (PLEG): container finished" podID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerID="073f03a885cbc288bfb004afad53a8e7166835ef38349a9b6211275b2c3ca905" exitCode=60
Oct 30 22:30:04.869200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.869112    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" event={"ID":"81c56d6f-0193-4922-a60f-b0aa5a71910b","Type":"ContainerDied","Data":"073f03a885cbc288bfb004afad53a8e7166835ef38349a9b6211275b2c3ca905"}
Oct 30 22:30:04.869404 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.869387    4282 scope.go:122] "RemoveContainer" containerID="073f03a885cbc288bfb004afad53a8e7166835ef38349a9b6211275b2c3ca905"
Oct 30 22:30:04.869720 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:04.869697    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj_odh-ai-gateway-infra(81c56d6f-0193-4922-a60f-b0aa5a71910b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b"
Oct 30 22:30:04.885686 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:04.885662    4282 scope.go:122] "RemoveContainer" containerID="7630fd96d8754673effcaf67bb30d8182e75cf93ded159ebe93758ccda0582d7"
Oct 30 22:30:05.881157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.881085    4282 scope.go:122] "RemoveContainer" containerID="6ffc2ad4ea260ed133f4aa576a1ba02da238aed415f709a94d37fa6d9e6880c7"
Oct 30 22:30:05.881585 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.881379    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6_odh-ai-gateway-infra(be9d37c2-8e42-411e-8e26-85a048947be6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" podUID="be9d37c2-8e42-411e-8e26-85a048947be6"
Oct 30 22:30:05.883163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.883138    4282 generic.go:356] "Generic (PLEG): container finished" podID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerID="29bbce338fe576f78835435c82c0978b53433be7de560f1d368d5d81a262f11c" exitCode=60
Oct 30 22:30:05.883289 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.883212    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" event={"ID":"0c2ce866-ccbc-4279-915e-cb75f1338701","Type":"ContainerDied","Data":"29bbce338fe576f78835435c82c0978b53433be7de560f1d368d5d81a262f11c"}
Oct 30 22:30:05.883354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.883295    4282 scope.go:122] "RemoveContainer" containerID="ca040500276cf0351cc371bd67e13a28de4feb6ca2e2738dda07315da773b0b1"
Oct 30 22:30:05.883530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.883515    4282 scope.go:122] "RemoveContainer" containerID="29bbce338fe576f78835435c82c0978b53433be7de560f1d368d5d81a262f11c"
Oct 30 22:30:05.883819 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.883796    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck_odh-ai-gateway-infra(0c2ce866-ccbc-4279-915e-cb75f1338701)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701"
Oct 30 22:30:05.885440 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.885423    4282 generic.go:356] "Generic (PLEG): container finished" podID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerID="dae5a9d71ee354ab296d238b3f15d9fe49d708b85ca39828e671d94d77408060" exitCode=60
Oct 30 22:30:05.885547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.885503    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" event={"ID":"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8","Type":"ContainerDied","Data":"dae5a9d71ee354ab296d238b3f15d9fe49d708b85ca39828e671d94d77408060"}
Oct 30 22:30:05.885735 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.885720    4282 scope.go:122] "RemoveContainer" containerID="dae5a9d71ee354ab296d238b3f15d9fe49d708b85ca39828e671d94d77408060"
Oct 30 22:30:05.885980 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.885957    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7_odh-ai-gateway-infra(122c7ccb-de31-4bba-9b54-bbc4ad6a32c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8"
Oct 30 22:30:05.887413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.887395    4282 generic.go:356] "Generic (PLEG): container finished" podID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerID="9e817ad5242dde8a2aaa03f6520fd84543c40d6b11ba5ad59f8d8bcf7d0b1e2c" exitCode=60
Oct 30 22:30:05.887519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.887489    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" event={"ID":"d2041356-82e2-4aaf-bf87-0ac90df55ba1","Type":"ContainerDied","Data":"9e817ad5242dde8a2aaa03f6520fd84543c40d6b11ba5ad59f8d8bcf7d0b1e2c"}
Oct 30 22:30:05.887694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.887680    4282 scope.go:122] "RemoveContainer" containerID="9e817ad5242dde8a2aaa03f6520fd84543c40d6b11ba5ad59f8d8bcf7d0b1e2c"
Oct 30 22:30:05.887951 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.887927    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb_odh-ai-gateway-infra(d2041356-82e2-4aaf-bf87-0ac90df55ba1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1"
Oct 30 22:30:05.889516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.889495    4282 generic.go:356] "Generic (PLEG): container finished" podID="28591d5d-928e-407f-b292-81ce32dc3393" containerID="238290933dd937fd967b31bfb1782570e3cf8c22f6fdc61fdff71b6d80c5e567" exitCode=60
Oct 30 22:30:05.889608 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.889573    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" event={"ID":"28591d5d-928e-407f-b292-81ce32dc3393","Type":"ContainerDied","Data":"238290933dd937fd967b31bfb1782570e3cf8c22f6fdc61fdff71b6d80c5e567"}
Oct 30 22:30:05.889766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.889748    4282 scope.go:122] "RemoveContainer" containerID="238290933dd937fd967b31bfb1782570e3cf8c22f6fdc61fdff71b6d80c5e567"
Oct 30 22:30:05.890045 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.890025    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj_odh-ai-gateway-infra(28591d5d-928e-407f-b292-81ce32dc3393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" podUID="28591d5d-928e-407f-b292-81ce32dc3393"
Oct 30 22:30:05.891354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.891334    4282 scope.go:122] "RemoveContainer" containerID="efd63847f335fd001f295b6e7f9a7297eac339189e6dfb38c3432389b956aeea"
Oct 30 22:30:05.891632 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.891612    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d_odh-ai-gateway-infra(14311e13-e401-442b-814e-4aa4e751e23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" podUID="14311e13-e401-442b-814e-4aa4e751e23c"
Oct 30 22:30:05.893992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.893970    4282 generic.go:356] "Generic (PLEG): container finished" podID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerID="54aa3ac6b4ce3e3893039ac435a6d2e1a22cc2803c9e767a315eeb7c08fc5955" exitCode=60
Oct 30 22:30:05.894069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.894042    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" event={"ID":"c43ffb1e-2508-4081-90e3-69febec2c84f","Type":"ContainerDied","Data":"54aa3ac6b4ce3e3893039ac435a6d2e1a22cc2803c9e767a315eeb7c08fc5955"}
Oct 30 22:30:05.894245 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.894220    4282 scope.go:122] "RemoveContainer" containerID="54aa3ac6b4ce3e3893039ac435a6d2e1a22cc2803c9e767a315eeb7c08fc5955"
Oct 30 22:30:05.894507 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.894484    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt_odh-ai-gateway-infra(c43ffb1e-2508-4081-90e3-69febec2c84f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f"
Oct 30 22:30:05.896114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.896096    4282 generic.go:356] "Generic (PLEG): container finished" podID="8410d5b0-f783-412b-8075-f83b710df94e" containerID="948194d042d784bed61ca9864b1a8b376b8f697b05f61ec9030d52d02556dc18" exitCode=60
Oct 30 22:30:05.896190 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.896165    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" event={"ID":"8410d5b0-f783-412b-8075-f83b710df94e","Type":"ContainerDied","Data":"948194d042d784bed61ca9864b1a8b376b8f697b05f61ec9030d52d02556dc18"}
Oct 30 22:30:05.896353 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.896340    4282 scope.go:122] "RemoveContainer" containerID="948194d042d784bed61ca9864b1a8b376b8f697b05f61ec9030d52d02556dc18"
Oct 30 22:30:05.896620 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.896601    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72_odh-ai-gateway-infra(8410d5b0-f783-412b-8075-f83b710df94e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" podUID="8410d5b0-f783-412b-8075-f83b710df94e"
Oct 30 22:30:05.898664 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.898639    4282 generic.go:356] "Generic (PLEG): container finished" podID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerID="a37e046269150ce2a182d9a86788d181e73d635cad9c54d77e771317eac49f18" exitCode=60
Oct 30 22:30:05.898761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.898735    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" event={"ID":"19855a1c-4f0c-478c-acf4-8eb39097fc92","Type":"ContainerDied","Data":"a37e046269150ce2a182d9a86788d181e73d635cad9c54d77e771317eac49f18"}
Oct 30 22:30:05.899035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.899019    4282 scope.go:122] "RemoveContainer" containerID="a37e046269150ce2a182d9a86788d181e73d635cad9c54d77e771317eac49f18"
Oct 30 22:30:05.899264 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.899243    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt_odh-ai-gateway-infra(19855a1c-4f0c-478c-acf4-8eb39097fc92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92"
Oct 30 22:30:05.899637 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.899618    4282 scope.go:122] "RemoveContainer" containerID="1c6b3ba0b45feef7c6e113027c9f1cfab581e70f4fe8286deaacfef9b8a62613"
Oct 30 22:30:05.900826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.900808    4282 scope.go:122] "RemoveContainer" containerID="073f03a885cbc288bfb004afad53a8e7166835ef38349a9b6211275b2c3ca905"
Oct 30 22:30:05.901063 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:05.901041    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj_odh-ai-gateway-infra(81c56d6f-0193-4922-a60f-b0aa5a71910b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b"
Oct 30 22:30:05.916514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.916454    4282 scope.go:122] "RemoveContainer" containerID="8bf62d9b65f582e3d714983e28fd11ca1a9888af32e87c39c8445d9ede1441d0"
Oct 30 22:30:05.931156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.931134    4282 scope.go:122] "RemoveContainer" containerID="2df0d7a491e00f8f3b495a699e7978706bba2f7546ab42bf31b67eb64e3436bc"
Oct 30 22:30:05.946054 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.946030    4282 scope.go:122] "RemoveContainer" containerID="5ead63034c6940b70afe46a35bbd00723c7d5cd9606666080a7b2cce5687f941"
Oct 30 22:30:05.966595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.966567    4282 scope.go:122] "RemoveContainer" containerID="bee67ff5f895e3ad50fe38ccd69eee2353bf33287235245c7f6c2b1a440e5307"
Oct 30 22:30:05.982724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:05.982703    4282 scope.go:122] "RemoveContainer" containerID="1c6f81ad1014b3e39e0408f42b2ff08bc743831ebc74ba5b09c7da9b893551ed"
Oct 30 22:30:06.913479 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:06.913425    4282 scope.go:122] "RemoveContainer" containerID="29bbce338fe576f78835435c82c0978b53433be7de560f1d368d5d81a262f11c"
Oct 30 22:30:06.913961 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:06.913836    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck_odh-ai-gateway-infra(0c2ce866-ccbc-4279-915e-cb75f1338701)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701"
Oct 30 22:30:06.915236 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:06.915213    4282 scope.go:122] "RemoveContainer" containerID="dae5a9d71ee354ab296d238b3f15d9fe49d708b85ca39828e671d94d77408060"
Oct 30 22:30:06.915554 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:06.915532    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7_odh-ai-gateway-infra(122c7ccb-de31-4bba-9b54-bbc4ad6a32c8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8"
Oct 30 22:30:06.916948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:06.916928    4282 scope.go:122] "RemoveContainer" containerID="9e817ad5242dde8a2aaa03f6520fd84543c40d6b11ba5ad59f8d8bcf7d0b1e2c"
Oct 30 22:30:06.917249 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:06.917228    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb_odh-ai-gateway-infra(d2041356-82e2-4aaf-bf87-0ac90df55ba1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1"
Oct 30 22:30:06.918766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:06.918736    4282 scope.go:122] "RemoveContainer" containerID="238290933dd937fd967b31bfb1782570e3cf8c22f6fdc61fdff71b6d80c5e567"
Oct 30 22:30:06.919145 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:06.919111    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj_odh-ai-gateway-infra(28591d5d-928e-407f-b292-81ce32dc3393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" podUID="28591d5d-928e-407f-b292-81ce32dc3393"
Oct 30 22:30:06.920395 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:06.920377    4282 scope.go:122] "RemoveContainer" containerID="54aa3ac6b4ce3e3893039ac435a6d2e1a22cc2803c9e767a315eeb7c08fc5955"
Oct 30 22:30:06.920679 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:06.920649    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt_odh-ai-gateway-infra(c43ffb1e-2508-4081-90e3-69febec2c84f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f"
Oct 30 22:30:06.922005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:06.921984    4282 scope.go:122] "RemoveContainer" containerID="948194d042d784bed61ca9864b1a8b376b8f697b05f61ec9030d52d02556dc18"
Oct 30 22:30:06.922232 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:06.922204    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72_odh-ai-gateway-infra(8410d5b0-f783-412b-8075-f83b710df94e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" podUID="8410d5b0-f783-412b-8075-f83b710df94e"
Oct 30 22:30:06.923516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:06.923496    4282 scope.go:122] "RemoveContainer" containerID="a37e046269150ce2a182d9a86788d181e73d635cad9c54d77e771317eac49f18"
Oct 30 22:30:06.923754 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:06.923733    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt_odh-ai-gateway-infra(19855a1c-4f0c-478c-acf4-8eb39097fc92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92"
Oct 30 22:30:15.468483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:15.468422    4282 scope.go:122] "RemoveContainer" containerID="6ffc2ad4ea260ed133f4aa576a1ba02da238aed415f709a94d37fa6d9e6880c7"
Oct 30 22:30:15.491800 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:15.491771    4282 scope.go:122] "RemoveContainer" containerID="073f03a885cbc288bfb004afad53a8e7166835ef38349a9b6211275b2c3ca905"
Oct 30 22:30:15.499424 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:15.493796    4282 scope.go:122] "RemoveContainer" containerID="948194d042d784bed61ca9864b1a8b376b8f697b05f61ec9030d52d02556dc18"
Oct 30 22:30:15.500308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:15.500270    4282 scope.go:122] "RemoveContainer" containerID="29bbce338fe576f78835435c82c0978b53433be7de560f1d368d5d81a262f11c"
Oct 30 22:30:15.997284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:15.997257    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" event={"ID":"8410d5b0-f783-412b-8075-f83b710df94e","Type":"ContainerStarted","Data":"90204171d1819c6746cb1b5d9b7550af0087118dae4799d5a4b60d6f8fdc2cd2"}
Oct 30 22:30:17.007513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.007406    4282 generic.go:356] "Generic (PLEG): container finished" podID="8410d5b0-f783-412b-8075-f83b710df94e" containerID="90204171d1819c6746cb1b5d9b7550af0087118dae4799d5a4b60d6f8fdc2cd2" exitCode=60
Oct 30 22:30:17.008931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.008188    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" event={"ID":"8410d5b0-f783-412b-8075-f83b710df94e","Type":"ContainerDied","Data":"90204171d1819c6746cb1b5d9b7550af0087118dae4799d5a4b60d6f8fdc2cd2"}
Oct 30 22:30:17.008931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.008301    4282 scope.go:122] "RemoveContainer" containerID="948194d042d784bed61ca9864b1a8b376b8f697b05f61ec9030d52d02556dc18"
Oct 30 22:30:17.027419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.009163    4282 scope.go:122] "RemoveContainer" containerID="90204171d1819c6746cb1b5d9b7550af0087118dae4799d5a4b60d6f8fdc2cd2"
Oct 30 22:30:17.066545 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:17.066291    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72_odh-ai-gateway-infra(8410d5b0-f783-412b-8075-f83b710df94e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" podUID="8410d5b0-f783-412b-8075-f83b710df94e"
Oct 30 22:30:17.073310 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.071588    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"]
Oct 30 22:30:17.077265 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.077233    4282 generic.go:356] "Generic (PLEG): container finished" podID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerID="ee1cf003287b26d23bfe978acb302b876789a94b840de54431e5595997e96606" exitCode=60
Oct 30 22:30:17.077405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.077324    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" event={"ID":"81c56d6f-0193-4922-a60f-b0aa5a71910b","Type":"ContainerDied","Data":"ee1cf003287b26d23bfe978acb302b876789a94b840de54431e5595997e96606"}
Oct 30 22:30:17.078320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.078295    4282 scope.go:122] "RemoveContainer" containerID="ee1cf003287b26d23bfe978acb302b876789a94b840de54431e5595997e96606"
Oct 30 22:30:17.078845 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:17.078807    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj_odh-ai-gateway-infra(81c56d6f-0193-4922-a60f-b0aa5a71910b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b"
Oct 30 22:30:17.080426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.080400    4282 generic.go:356] "Generic (PLEG): container finished" podID="be9d37c2-8e42-411e-8e26-85a048947be6" containerID="9fcb9bbb4712c871d0ccc6d1872e7a1011cc369e2e45c374da0027c67bbbc034" exitCode=60
Oct 30 22:30:17.080579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.080524    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" event={"ID":"be9d37c2-8e42-411e-8e26-85a048947be6","Type":"ContainerDied","Data":"9fcb9bbb4712c871d0ccc6d1872e7a1011cc369e2e45c374da0027c67bbbc034"}
Oct 30 22:30:17.081443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.081130    4282 scope.go:122] "RemoveContainer" containerID="9fcb9bbb4712c871d0ccc6d1872e7a1011cc369e2e45c374da0027c67bbbc034"
Oct 30 22:30:17.081569 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:17.081543    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6_odh-ai-gateway-infra(be9d37c2-8e42-411e-8e26-85a048947be6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" podUID="be9d37c2-8e42-411e-8e26-85a048947be6"
Oct 30 22:30:17.083582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.083553    4282 generic.go:356] "Generic (PLEG): container finished" podID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerID="d5b95241b19c9c98244ad958ccc38f24ca18850c084209c80a73f3cf7301e551" exitCode=60
Oct 30 22:30:17.083671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.083598    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" event={"ID":"0c2ce866-ccbc-4279-915e-cb75f1338701","Type":"ContainerDied","Data":"d5b95241b19c9c98244ad958ccc38f24ca18850c084209c80a73f3cf7301e551"}
Oct 30 22:30:17.083955 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.083941    4282 scope.go:122] "RemoveContainer" containerID="d5b95241b19c9c98244ad958ccc38f24ca18850c084209c80a73f3cf7301e551"
Oct 30 22:30:17.084210 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:17.084189    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck_odh-ai-gateway-infra(0c2ce866-ccbc-4279-915e-cb75f1338701)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701"
Oct 30 22:30:17.091368 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.091350    4282 scope.go:122] "RemoveContainer" containerID="073f03a885cbc288bfb004afad53a8e7166835ef38349a9b6211275b2c3ca905"
Oct 30 22:30:17.109395 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.109371    4282 scope.go:122] "RemoveContainer" containerID="6ffc2ad4ea260ed133f4aa576a1ba02da238aed415f709a94d37fa6d9e6880c7"
Oct 30 22:30:17.123891 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.123871    4282 scope.go:122] "RemoveContainer" containerID="29bbce338fe576f78835435c82c0978b53433be7de560f1d368d5d81a262f11c"
Oct 30 22:30:17.467674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.467650    4282 scope.go:122] "RemoveContainer" containerID="9e817ad5242dde8a2aaa03f6520fd84543c40d6b11ba5ad59f8d8bcf7d0b1e2c"
Oct 30 22:30:17.467869 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.467851    4282 scope.go:122] "RemoveContainer" containerID="54aa3ac6b4ce3e3893039ac435a6d2e1a22cc2803c9e767a315eeb7c08fc5955"
Oct 30 22:30:17.468199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:17.468176    4282 scope.go:122] "RemoveContainer" containerID="efd63847f335fd001f295b6e7f9a7297eac339189e6dfb38c3432389b956aeea"
Oct 30 22:30:18.095572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.095444    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" event={"ID":"d2041356-82e2-4aaf-bf87-0ac90df55ba1","Type":"ContainerStarted","Data":"0621c9ef21cab5d0bda8035fbfd24e03aa721d7349bc04413bd679423b6a8f6d"}
Oct 30 22:30:18.097485 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.097450    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" event={"ID":"14311e13-e401-442b-814e-4aa4e751e23c","Type":"ContainerStarted","Data":"c9332521209abeb3c8470e0cad73ab6dff8bcda58c11062396b299366775e06e"}
Oct 30 22:30:18.099534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.099509    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" event={"ID":"c43ffb1e-2508-4081-90e3-69febec2c84f","Type":"ContainerStarted","Data":"c7cac0219c3c92b759689e93f49166f61ebcf10ed01d63b4b347e14beeb96d46"}
Oct 30 22:30:18.112829 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.112806    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"]
Oct 30 22:30:18.141591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.141523    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"]
Oct 30 22:30:18.154974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.153372    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"]
Oct 30 22:30:18.344057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.344028    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:18.421624 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.421597    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:18.460204 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.459640    4282 scope.go:122] "RemoveContainer" containerID="a37e046269150ce2a182d9a86788d181e73d635cad9c54d77e771317eac49f18"
Oct 30 22:30:18.467372 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.463750    4282 scope.go:122] "RemoveContainer" containerID="dae5a9d71ee354ab296d238b3f15d9fe49d708b85ca39828e671d94d77408060"
Oct 30 22:30:18.498124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.498088    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c56d6f-0193-4922-a60f-b0aa5a71910b-kube-api-access-2qlwl\" (UniqueName: \"kubernetes.io/projected/81c56d6f-0193-4922-a60f-b0aa5a71910b-kube-api-access-2qlwl\") pod \"81c56d6f-0193-4922-a60f-b0aa5a71910b\" (UID: \"81c56d6f-0193-4922-a60f-b0aa5a71910b\") "
Oct 30 22:30:18.499755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.499725    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8410d5b0-f783-412b-8075-f83b710df94e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8410d5b0-f783-412b-8075-f83b710df94e-maas-api-service-ca\") pod \"8410d5b0-f783-412b-8075-f83b710df94e\" (UID: \"8410d5b0-f783-412b-8075-f83b710df94e\") "
Oct 30 22:30:18.508862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.508823    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81c56d6f-0193-4922-a60f-b0aa5a71910b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c56d6f-0193-4922-a60f-b0aa5a71910b-maas-api-service-ca\") pod \"81c56d6f-0193-4922-a60f-b0aa5a71910b\" (UID: \"81c56d6f-0193-4922-a60f-b0aa5a71910b\") "
Oct 30 22:30:18.515397 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.515361    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8410d5b0-f783-412b-8075-f83b710df94e-kube-api-access-xsw4b\" (UniqueName: \"kubernetes.io/projected/8410d5b0-f783-412b-8075-f83b710df94e-kube-api-access-xsw4b\") pod \"8410d5b0-f783-412b-8075-f83b710df94e\" (UID: \"8410d5b0-f783-412b-8075-f83b710df94e\") "
Oct 30 22:30:18.516936 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.513124    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8410d5b0-f783-412b-8075-f83b710df94e-maas-api-service-ca" pod "8410d5b0-f783-412b-8075-f83b710df94e" (UID: "8410d5b0-f783-412b-8075-f83b710df94e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:18.517103 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.515018    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c56d6f-0193-4922-a60f-b0aa5a71910b-maas-api-service-ca" pod "81c56d6f-0193-4922-a60f-b0aa5a71910b" (UID: "81c56d6f-0193-4922-a60f-b0aa5a71910b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:18.559144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.559074    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c56d6f-0193-4922-a60f-b0aa5a71910b-kube-api-access-2qlwl" pod "81c56d6f-0193-4922-a60f-b0aa5a71910b" (UID: "81c56d6f-0193-4922-a60f-b0aa5a71910b"). InnerVolumeSpecName "kube-api-access-2qlwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.583090 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.583051    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8410d5b0-f783-412b-8075-f83b710df94e-kube-api-access-xsw4b" pod "8410d5b0-f783-412b-8075-f83b710df94e" (UID: "8410d5b0-f783-412b-8075-f83b710df94e"). InnerVolumeSpecName "kube-api-access-xsw4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.618927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.618868    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qlwl\" (UniqueName: \"kubernetes.io/projected/81c56d6f-0193-4922-a60f-b0aa5a71910b-kube-api-access-2qlwl\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.618927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.618891    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8410d5b0-f783-412b-8075-f83b710df94e-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.618927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.618907    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c56d6f-0193-4922-a60f-b0aa5a71910b-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.618927 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.618917    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsw4b\" (UniqueName: \"kubernetes.io/projected/8410d5b0-f783-412b-8075-f83b710df94e-kube-api-access-xsw4b\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.629991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.629969    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:18.646398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.646380    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:18.720309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.720275    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2ce866-ccbc-4279-915e-cb75f1338701-kube-api-access-zmjbq\" (UniqueName: \"kubernetes.io/projected/0c2ce866-ccbc-4279-915e-cb75f1338701-kube-api-access-zmjbq\") pod \"0c2ce866-ccbc-4279-915e-cb75f1338701\" (UID: \"0c2ce866-ccbc-4279-915e-cb75f1338701\") "
Oct 30 22:30:18.721001 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.720955    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2ce866-ccbc-4279-915e-cb75f1338701-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ce866-ccbc-4279-915e-cb75f1338701-maas-api-service-ca\") pod \"0c2ce866-ccbc-4279-915e-cb75f1338701\" (UID: \"0c2ce866-ccbc-4279-915e-cb75f1338701\") "
Oct 30 22:30:18.722622 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.722587    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2ce866-ccbc-4279-915e-cb75f1338701-maas-api-service-ca" pod "0c2ce866-ccbc-4279-915e-cb75f1338701" (UID: "0c2ce866-ccbc-4279-915e-cb75f1338701"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:18.754537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.754494    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2ce866-ccbc-4279-915e-cb75f1338701-kube-api-access-zmjbq" pod "0c2ce866-ccbc-4279-915e-cb75f1338701" (UID: "0c2ce866-ccbc-4279-915e-cb75f1338701"). InnerVolumeSpecName "kube-api-access-zmjbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.823035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.822999    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be9d37c2-8e42-411e-8e26-85a048947be6-kube-api-access-rdkr4\" (UniqueName: \"kubernetes.io/projected/be9d37c2-8e42-411e-8e26-85a048947be6-kube-api-access-rdkr4\") pod \"be9d37c2-8e42-411e-8e26-85a048947be6\" (UID: \"be9d37c2-8e42-411e-8e26-85a048947be6\") "
Oct 30 22:30:18.823208 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.823196    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be9d37c2-8e42-411e-8e26-85a048947be6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9d37c2-8e42-411e-8e26-85a048947be6-maas-api-service-ca\") pod \"be9d37c2-8e42-411e-8e26-85a048947be6\" (UID: \"be9d37c2-8e42-411e-8e26-85a048947be6\") "
Oct 30 22:30:18.823576 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.823550    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9d37c2-8e42-411e-8e26-85a048947be6-maas-api-service-ca" pod "be9d37c2-8e42-411e-8e26-85a048947be6" (UID: "be9d37c2-8e42-411e-8e26-85a048947be6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:18.823912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.823890    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmjbq\" (UniqueName: \"kubernetes.io/projected/0c2ce866-ccbc-4279-915e-cb75f1338701-kube-api-access-zmjbq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.823991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.823918    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9d37c2-8e42-411e-8e26-85a048947be6-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.823991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.823933    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2ce866-ccbc-4279-915e-cb75f1338701-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.825271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.825251    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be9d37c2-8e42-411e-8e26-85a048947be6-kube-api-access-rdkr4" pod "be9d37c2-8e42-411e-8e26-85a048947be6" (UID: "be9d37c2-8e42-411e-8e26-85a048947be6"). InnerVolumeSpecName "kube-api-access-rdkr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:18.925213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:18.925185    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdkr4\" (UniqueName: \"kubernetes.io/projected/be9d37c2-8e42-411e-8e26-85a048947be6-kube-api-access-rdkr4\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.114198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.114174    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"
Oct 30 22:30:19.114731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.114165    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj" event={"ID":"81c56d6f-0193-4922-a60f-b0aa5a71910b","Type":"ContainerDied","Data":"5c1475dbe639f9fd163146794d481846cabc2c90cd9ee152dc5e0d67dda36a33"}
Oct 30 22:30:19.114731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.114350    4282 scope.go:122] "RemoveContainer" containerID="ee1cf003287b26d23bfe978acb302b876789a94b840de54431e5595997e96606"
Oct 30 22:30:19.115997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.115942    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"
Oct 30 22:30:19.115997 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.115976    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6" event={"ID":"be9d37c2-8e42-411e-8e26-85a048947be6","Type":"ContainerDied","Data":"e7a056e11c69a159442d06c08689dd752b0fc82f806f82c30b9939d7f1d17019"}
Oct 30 22:30:19.118531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.118508    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"
Oct 30 22:30:19.131213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.131166    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck" event={"ID":"0c2ce866-ccbc-4279-915e-cb75f1338701","Type":"ContainerDied","Data":"7ba770097f2b21e3b943381fb9fe2cb0cd8d8a258890177681892f984be925ed"}
Oct 30 22:30:19.160580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.160543    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" event={"ID":"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8","Type":"ContainerStarted","Data":"8eed732fb70463a1afb0ef750c62f3a47d45b5abc22b11173dfbc3dc66ed07ef"}
Oct 30 22:30:19.172168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.172135    4282 generic.go:356] "Generic (PLEG): container finished" podID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerID="0621c9ef21cab5d0bda8035fbfd24e03aa721d7349bc04413bd679423b6a8f6d" exitCode=60
Oct 30 22:30:19.173663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.173631    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" event={"ID":"d2041356-82e2-4aaf-bf87-0ac90df55ba1","Type":"ContainerDied","Data":"0621c9ef21cab5d0bda8035fbfd24e03aa721d7349bc04413bd679423b6a8f6d"}
Oct 30 22:30:19.218427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.192144    4282 generic.go:356] "Generic (PLEG): container finished" podID="14311e13-e401-442b-814e-4aa4e751e23c" containerID="c9332521209abeb3c8470e0cad73ab6dff8bcda58c11062396b299366775e06e" exitCode=60
Oct 30 22:30:19.218427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.214733    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"]
Oct 30 22:30:19.218427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.215456    4282 scope.go:122] "RemoveContainer" containerID="0621c9ef21cab5d0bda8035fbfd24e03aa721d7349bc04413bd679423b6a8f6d"
Oct 30 22:30:19.218427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.215568    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" event={"ID":"14311e13-e401-442b-814e-4aa4e751e23c","Type":"ContainerDied","Data":"c9332521209abeb3c8470e0cad73ab6dff8bcda58c11062396b299366775e06e"}
Oct 30 22:30:19.218427 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:19.216075    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb_odh-ai-gateway-infra(d2041356-82e2-4aaf-bf87-0ac90df55ba1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1"
Oct 30 22:30:19.218427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.217818    4282 scope.go:122] "RemoveContainer" containerID="c9332521209abeb3c8470e0cad73ab6dff8bcda58c11062396b299366775e06e"
Oct 30 22:30:19.218427 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:19.218376    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d_odh-ai-gateway-infra(14311e13-e401-442b-814e-4aa4e751e23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" podUID="14311e13-e401-442b-814e-4aa4e751e23c"
Oct 30 22:30:19.221311 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.221266    4282 scope.go:122] "RemoveContainer" containerID="9fcb9bbb4712c871d0ccc6d1872e7a1011cc369e2e45c374da0027c67bbbc034"
Oct 30 22:30:19.225881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.225846    4282 generic.go:356] "Generic (PLEG): container finished" podID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerID="c7cac0219c3c92b759689e93f49166f61ebcf10ed01d63b4b347e14beeb96d46" exitCode=60
Oct 30 22:30:19.225992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.225964    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" event={"ID":"c43ffb1e-2508-4081-90e3-69febec2c84f","Type":"ContainerDied","Data":"c7cac0219c3c92b759689e93f49166f61ebcf10ed01d63b4b347e14beeb96d46"}
Oct 30 22:30:19.226315 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.226299    4282 scope.go:122] "RemoveContainer" containerID="c7cac0219c3c92b759689e93f49166f61ebcf10ed01d63b4b347e14beeb96d46"
Oct 30 22:30:19.226770 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:19.226749    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt_odh-ai-gateway-infra(c43ffb1e-2508-4081-90e3-69febec2c84f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f"
Oct 30 22:30:19.232536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.228586    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"
Oct 30 22:30:19.232536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.228554    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72" event={"ID":"8410d5b0-f783-412b-8075-f83b710df94e","Type":"ContainerDied","Data":"87617039754a24ddd29683b299c60b25436db7f730a1b69e5b2eaf592fb63634"}
Oct 30 22:30:19.232536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.229802    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"]
Oct 30 22:30:19.232536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.231021    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" event={"ID":"19855a1c-4f0c-478c-acf4-8eb39097fc92","Type":"ContainerStarted","Data":"72f2ff2f47a3f8e2c9e59d7c02f5cc1aab48abe732cbdaf19c83f2081d50b962"}
Oct 30 22:30:19.236070 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.235667    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-120cb1-29889990-6rshj"]
Oct 30 22:30:19.246445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.245512    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"]
Oct 30 22:30:19.246445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.246414    4282 scope.go:122] "RemoveContainer" containerID="d5b95241b19c9c98244ad958ccc38f24ca18850c084209c80a73f3cf7301e551"
Oct 30 22:30:19.258815 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.258793    4282 scope.go:122] "RemoveContainer" containerID="9e817ad5242dde8a2aaa03f6520fd84543c40d6b11ba5ad59f8d8bcf7d0b1e2c"
Oct 30 22:30:19.271968 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.271939    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"]
Oct 30 22:30:19.278439 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.278409    4282 scope.go:122] "RemoveContainer" containerID="efd63847f335fd001f295b6e7f9a7297eac339189e6dfb38c3432389b956aeea"
Oct 30 22:30:19.300159 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.300135    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"]
Oct 30 22:30:19.307536 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.307516    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1529dc-29889990-wl8c6"]
Oct 30 22:30:19.325774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.325751    4282 scope.go:122] "RemoveContainer" containerID="54aa3ac6b4ce3e3893039ac435a6d2e1a22cc2803c9e767a315eeb7c08fc5955"
Oct 30 22:30:19.341298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.341280    4282 scope.go:122] "RemoveContainer" containerID="90204171d1819c6746cb1b5d9b7550af0087118dae4799d5a4b60d6f8fdc2cd2"
Oct 30 22:30:19.343106 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.343083    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"]
Oct 30 22:30:19.349391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.349367    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5f192-29889990-nkcck"]
Oct 30 22:30:19.359117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.359097    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"]
Oct 30 22:30:19.365564 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.365544    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a39158-29889990-fbc72"]
Oct 30 22:30:19.457210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.457181    4282 scope.go:122] "RemoveContainer" containerID="238290933dd937fd967b31bfb1782570e3cf8c22f6fdc61fdff71b6d80c5e567"
Oct 30 22:30:19.470346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.470320    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" path="/var/lib/kubelet/pods/0c2ce866-ccbc-4279-915e-cb75f1338701/volumes"
Oct 30 22:30:19.470868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.470845    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" path="/var/lib/kubelet/pods/81c56d6f-0193-4922-a60f-b0aa5a71910b/volumes"
Oct 30 22:30:19.471277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.471262    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8410d5b0-f783-412b-8075-f83b710df94e" path="/var/lib/kubelet/pods/8410d5b0-f783-412b-8075-f83b710df94e/volumes"
Oct 30 22:30:19.472164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:19.472144    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" path="/var/lib/kubelet/pods/be9d37c2-8e42-411e-8e26-85a048947be6/volumes"
Oct 30 22:30:20.229545 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.229516    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"]
Oct 30 22:30:20.243175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.243143    4282 generic.go:356] "Generic (PLEG): container finished" podID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerID="72f2ff2f47a3f8e2c9e59d7c02f5cc1aab48abe732cbdaf19c83f2081d50b962" exitCode=60
Oct 30 22:30:20.243350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.243214    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" event={"ID":"19855a1c-4f0c-478c-acf4-8eb39097fc92","Type":"ContainerDied","Data":"72f2ff2f47a3f8e2c9e59d7c02f5cc1aab48abe732cbdaf19c83f2081d50b962"}
Oct 30 22:30:20.243350 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.243277    4282 scope.go:122] "RemoveContainer" containerID="a37e046269150ce2a182d9a86788d181e73d635cad9c54d77e771317eac49f18"
Oct 30 22:30:20.243789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.243773    4282 scope.go:122] "RemoveContainer" containerID="72f2ff2f47a3f8e2c9e59d7c02f5cc1aab48abe732cbdaf19c83f2081d50b962"
Oct 30 22:30:20.244628 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:20.244245    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt_odh-ai-gateway-infra(19855a1c-4f0c-478c-acf4-8eb39097fc92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92"
Oct 30 22:30:20.249930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.249906    4282 generic.go:356] "Generic (PLEG): container finished" podID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerID="8eed732fb70463a1afb0ef750c62f3a47d45b5abc22b11173dfbc3dc66ed07ef" exitCode=60
Oct 30 22:30:20.250051 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.250021    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" event={"ID":"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8","Type":"ContainerDied","Data":"8eed732fb70463a1afb0ef750c62f3a47d45b5abc22b11173dfbc3dc66ed07ef"}
Oct 30 22:30:20.254861 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.254833    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" event={"ID":"28591d5d-928e-407f-b292-81ce32dc3393","Type":"ContainerStarted","Data":"2c8673c4c14241b05057ca361888516b4ff8cf8fbaa663b2dff2c5ed6dc77857"}
Oct 30 22:30:20.273133 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.273105    4282 scope.go:122] "RemoveContainer" containerID="dae5a9d71ee354ab296d238b3f15d9fe49d708b85ca39828e671d94d77408060"
Oct 30 22:30:20.332191 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.332164    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"]
Oct 30 22:30:20.574442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.574416    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:20.580986 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.580952    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:20.720777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.711998    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2041356-82e2-4aaf-bf87-0ac90df55ba1-kube-api-access-wdwx5\" (UniqueName: \"kubernetes.io/projected/d2041356-82e2-4aaf-bf87-0ac90df55ba1-kube-api-access-wdwx5\") pod \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\" (UID: \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\") "
Oct 30 22:30:20.720777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.712163    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2041356-82e2-4aaf-bf87-0ac90df55ba1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2041356-82e2-4aaf-bf87-0ac90df55ba1-maas-api-service-ca\") pod \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\" (UID: \"d2041356-82e2-4aaf-bf87-0ac90df55ba1\") "
Oct 30 22:30:20.720777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.712282    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14311e13-e401-442b-814e-4aa4e751e23c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14311e13-e401-442b-814e-4aa4e751e23c-maas-api-service-ca\") pod \"14311e13-e401-442b-814e-4aa4e751e23c\" (UID: \"14311e13-e401-442b-814e-4aa4e751e23c\") "
Oct 30 22:30:20.720777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.712512    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14311e13-e401-442b-814e-4aa4e751e23c-kube-api-access-cxncj\" (UniqueName: \"kubernetes.io/projected/14311e13-e401-442b-814e-4aa4e751e23c-kube-api-access-cxncj\") pod \"14311e13-e401-442b-814e-4aa4e751e23c\" (UID: \"14311e13-e401-442b-814e-4aa4e751e23c\") "
Oct 30 22:30:20.720777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.715014    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2041356-82e2-4aaf-bf87-0ac90df55ba1-maas-api-service-ca" pod "d2041356-82e2-4aaf-bf87-0ac90df55ba1" (UID: "d2041356-82e2-4aaf-bf87-0ac90df55ba1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.720777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.716226    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14311e13-e401-442b-814e-4aa4e751e23c-maas-api-service-ca" pod "14311e13-e401-442b-814e-4aa4e751e23c" (UID: "14311e13-e401-442b-814e-4aa4e751e23c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.720777 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.719307    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2041356-82e2-4aaf-bf87-0ac90df55ba1-kube-api-access-wdwx5" pod "d2041356-82e2-4aaf-bf87-0ac90df55ba1" (UID: "d2041356-82e2-4aaf-bf87-0ac90df55ba1"). InnerVolumeSpecName "kube-api-access-wdwx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.723595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.722722    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14311e13-e401-442b-814e-4aa4e751e23c-kube-api-access-cxncj" pod "14311e13-e401-442b-814e-4aa4e751e23c" (UID: "14311e13-e401-442b-814e-4aa4e751e23c"). InnerVolumeSpecName "kube-api-access-cxncj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.726496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.726115    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwx5\" (UniqueName: \"kubernetes.io/projected/d2041356-82e2-4aaf-bf87-0ac90df55ba1-kube-api-access-wdwx5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.726496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.726147    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2041356-82e2-4aaf-bf87-0ac90df55ba1-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.726496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.726161    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14311e13-e401-442b-814e-4aa4e751e23c-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.734589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.733050    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:20.748710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.748645    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:20.827415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.827384    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-maas-api-service-ca\") pod \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\" (UID: \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\") "
Oct 30 22:30:20.827592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.827569    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c43ffb1e-2508-4081-90e3-69febec2c84f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffb1e-2508-4081-90e3-69febec2c84f-maas-api-service-ca\") pod \"c43ffb1e-2508-4081-90e3-69febec2c84f\" (UID: \"c43ffb1e-2508-4081-90e3-69febec2c84f\") "
Oct 30 22:30:20.827634 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.827593    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43ffb1e-2508-4081-90e3-69febec2c84f-kube-api-access-2sc4h\" (UniqueName: \"kubernetes.io/projected/c43ffb1e-2508-4081-90e3-69febec2c84f-kube-api-access-2sc4h\") pod \"c43ffb1e-2508-4081-90e3-69febec2c84f\" (UID: \"c43ffb1e-2508-4081-90e3-69febec2c84f\") "
Oct 30 22:30:20.827690 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.827668    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-kube-api-access-hk62d\" (UniqueName: \"kubernetes.io/projected/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-kube-api-access-hk62d\") pod \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\" (UID: \"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8\") "
Oct 30 22:30:20.827888 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.827857    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-maas-api-service-ca" pod "122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" (UID: "122c7ccb-de31-4bba-9b54-bbc4ad6a32c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.827979 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.827929    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c43ffb1e-2508-4081-90e3-69febec2c84f-maas-api-service-ca" pod "c43ffb1e-2508-4081-90e3-69febec2c84f" (UID: "c43ffb1e-2508-4081-90e3-69febec2c84f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.828274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.828243    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.828274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.828266    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c43ffb1e-2508-4081-90e3-69febec2c84f-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.828274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.828280    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxncj\" (UniqueName: \"kubernetes.io/projected/14311e13-e401-442b-814e-4aa4e751e23c-kube-api-access-cxncj\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.830020 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.829996    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43ffb1e-2508-4081-90e3-69febec2c84f-kube-api-access-2sc4h" pod "c43ffb1e-2508-4081-90e3-69febec2c84f" (UID: "c43ffb1e-2508-4081-90e3-69febec2c84f"). InnerVolumeSpecName "kube-api-access-2sc4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.830119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.830016    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-kube-api-access-hk62d" pod "122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" (UID: "122c7ccb-de31-4bba-9b54-bbc4ad6a32c8"). InnerVolumeSpecName "kube-api-access-hk62d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.930017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.929989    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk62d\" (UniqueName: \"kubernetes.io/projected/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8-kube-api-access-hk62d\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.930017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:20.930012    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sc4h\" (UniqueName: \"kubernetes.io/projected/c43ffb1e-2508-4081-90e3-69febec2c84f-kube-api-access-2sc4h\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:21.269573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.269542    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"
Oct 30 22:30:21.270085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.269592    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7" event={"ID":"122c7ccb-de31-4bba-9b54-bbc4ad6a32c8","Type":"ContainerDied","Data":"dcf94176f4cc9a8426315a77bb6ea45717dd4e8715e23d81f822651c77360978"}
Oct 30 22:30:21.270085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.269690    4282 scope.go:122] "RemoveContainer" containerID="8eed732fb70463a1afb0ef750c62f3a47d45b5abc22b11173dfbc3dc66ed07ef"
Oct 30 22:30:21.271180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.271165    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"
Oct 30 22:30:21.271295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.271185    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb" event={"ID":"d2041356-82e2-4aaf-bf87-0ac90df55ba1","Type":"ContainerDied","Data":"775a1aacc9eafc8bd8447c12459ceb9c1a89b48222a4e3fc8015855f1f50add4"}
Oct 30 22:30:21.273684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.273657    4282 generic.go:356] "Generic (PLEG): container finished" podID="28591d5d-928e-407f-b292-81ce32dc3393" containerID="2c8673c4c14241b05057ca361888516b4ff8cf8fbaa663b2dff2c5ed6dc77857" exitCode=60
Oct 30 22:30:21.273803 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.273737    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" event={"ID":"28591d5d-928e-407f-b292-81ce32dc3393","Type":"ContainerDied","Data":"2c8673c4c14241b05057ca361888516b4ff8cf8fbaa663b2dff2c5ed6dc77857"}
Oct 30 22:30:21.274232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.274209    4282 scope.go:122] "RemoveContainer" containerID="2c8673c4c14241b05057ca361888516b4ff8cf8fbaa663b2dff2c5ed6dc77857"
Oct 30 22:30:21.274632 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:21.274607    4282 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj_odh-ai-gateway-infra(28591d5d-928e-407f-b292-81ce32dc3393)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" podUID="28591d5d-928e-407f-b292-81ce32dc3393"
Oct 30 22:30:21.275512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.275455    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"
Oct 30 22:30:21.275720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.275507    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d" event={"ID":"14311e13-e401-442b-814e-4aa4e751e23c","Type":"ContainerDied","Data":"5b7e7055c4db0d507886a1f08846a19824eee208e29b5cb76fef57dd06e5e253"}
Oct 30 22:30:21.277811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.277787    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"
Oct 30 22:30:21.277929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.277867    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt" event={"ID":"c43ffb1e-2508-4081-90e3-69febec2c84f","Type":"ContainerDied","Data":"fe96e488f464c180311791ae3e6661799823bdcc85ab004778f7fdcf64396b0d"}
Oct 30 22:30:21.287197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.287170    4282 scope.go:122] "RemoveContainer" containerID="0621c9ef21cab5d0bda8035fbfd24e03aa721d7349bc04413bd679423b6a8f6d"
Oct 30 22:30:21.305523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.305503    4282 scope.go:122] "RemoveContainer" containerID="238290933dd937fd967b31bfb1782570e3cf8c22f6fdc61fdff71b6d80c5e567"
Oct 30 22:30:21.312544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.312514    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"]
Oct 30 22:30:21.324055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.324025    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b9d868-29889990-dqmpt"]
Oct 30 22:30:21.324904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.324879    4282 scope.go:122] "RemoveContainer" containerID="c9332521209abeb3c8470e0cad73ab6dff8bcda58c11062396b299366775e06e"
Oct 30 22:30:21.335820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.335802    4282 scope.go:122] "RemoveContainer" containerID="c7cac0219c3c92b759689e93f49166f61ebcf10ed01d63b4b347e14beeb96d46"
Oct 30 22:30:21.354635 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.354610    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"]
Oct 30 22:30:21.387442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.387410    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"]
Oct 30 22:30:21.398511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.395044    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a1ea1e-29889990-qnrw7"]
Oct 30 22:30:21.407193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.407168    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"]
Oct 30 22:30:21.412361 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.412332    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-06ac2e-29889990-fz9rb"]
Oct 30 22:30:21.436023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.435990    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"]
Oct 30 22:30:21.442618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.442593    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-28b4bb-29889990-kl52d"]
Oct 30 22:30:21.449178 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.449158    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:21.480166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.480120    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" path="/var/lib/kubelet/pods/122c7ccb-de31-4bba-9b54-bbc4ad6a32c8/volumes"
Oct 30 22:30:21.480923 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.480895    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14311e13-e401-442b-814e-4aa4e751e23c" path="/var/lib/kubelet/pods/14311e13-e401-442b-814e-4aa4e751e23c/volumes"
Oct 30 22:30:21.481582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.481561    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" path="/var/lib/kubelet/pods/c43ffb1e-2508-4081-90e3-69febec2c84f/volumes"
Oct 30 22:30:21.482620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.482597    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" path="/var/lib/kubelet/pods/d2041356-82e2-4aaf-bf87-0ac90df55ba1/volumes"
Oct 30 22:30:21.564503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.564366    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19855a1c-4f0c-478c-acf4-8eb39097fc92-kube-api-access-6vdjl\" (UniqueName: \"kubernetes.io/projected/19855a1c-4f0c-478c-acf4-8eb39097fc92-kube-api-access-6vdjl\") pod \"19855a1c-4f0c-478c-acf4-8eb39097fc92\" (UID: \"19855a1c-4f0c-478c-acf4-8eb39097fc92\") "
Oct 30 22:30:21.564884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.564855    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19855a1c-4f0c-478c-acf4-8eb39097fc92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19855a1c-4f0c-478c-acf4-8eb39097fc92-maas-api-service-ca\") pod \"19855a1c-4f0c-478c-acf4-8eb39097fc92\" (UID: \"19855a1c-4f0c-478c-acf4-8eb39097fc92\") "
Oct 30 22:30:21.585942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.585903    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19855a1c-4f0c-478c-acf4-8eb39097fc92-maas-api-service-ca" pod "19855a1c-4f0c-478c-acf4-8eb39097fc92" (UID: "19855a1c-4f0c-478c-acf4-8eb39097fc92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:21.591028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.590992    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19855a1c-4f0c-478c-acf4-8eb39097fc92-kube-api-access-6vdjl" pod "19855a1c-4f0c-478c-acf4-8eb39097fc92" (UID: "19855a1c-4f0c-478c-acf4-8eb39097fc92"). InnerVolumeSpecName "kube-api-access-6vdjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:21.675302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.675273    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdjl\" (UniqueName: \"kubernetes.io/projected/19855a1c-4f0c-478c-acf4-8eb39097fc92-kube-api-access-6vdjl\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:21.675302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:21.675302    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19855a1c-4f0c-478c-acf4-8eb39097fc92-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:22.301110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.301015    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"
Oct 30 22:30:22.301110 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.301055    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt" event={"ID":"19855a1c-4f0c-478c-acf4-8eb39097fc92","Type":"ContainerDied","Data":"216a56b2bfaef88f633b9ae9652c936d6021395bb9263210ed3f6f211a990351"}
Oct 30 22:30:22.301649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.301129    4282 scope.go:122] "RemoveContainer" containerID="72f2ff2f47a3f8e2c9e59d7c02f5cc1aab48abe732cbdaf19c83f2081d50b962"
Oct 30 22:30:22.339042 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.339004    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"]
Oct 30 22:30:22.344814 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.344787    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ceb36a-29889990-qsfwt"]
Oct 30 22:30:22.531354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.531332    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:22.590799 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.590725    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28591d5d-928e-407f-b292-81ce32dc3393-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28591d5d-928e-407f-b292-81ce32dc3393-maas-api-service-ca\") pod \"28591d5d-928e-407f-b292-81ce32dc3393\" (UID: \"28591d5d-928e-407f-b292-81ce32dc3393\") "
Oct 30 22:30:22.590974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.590803    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28591d5d-928e-407f-b292-81ce32dc3393-kube-api-access-7549d\" (UniqueName: \"kubernetes.io/projected/28591d5d-928e-407f-b292-81ce32dc3393-kube-api-access-7549d\") pod \"28591d5d-928e-407f-b292-81ce32dc3393\" (UID: \"28591d5d-928e-407f-b292-81ce32dc3393\") "
Oct 30 22:30:22.591192 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.591160    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28591d5d-928e-407f-b292-81ce32dc3393-maas-api-service-ca" pod "28591d5d-928e-407f-b292-81ce32dc3393" (UID: "28591d5d-928e-407f-b292-81ce32dc3393"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:22.591732 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.591710    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28591d5d-928e-407f-b292-81ce32dc3393-maas-api-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:22.593903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.593867    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28591d5d-928e-407f-b292-81ce32dc3393-kube-api-access-7549d" pod "28591d5d-928e-407f-b292-81ce32dc3393" (UID: "28591d5d-928e-407f-b292-81ce32dc3393"). InnerVolumeSpecName "kube-api-access-7549d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:22.694684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:22.694657    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7549d\" (UniqueName: \"kubernetes.io/projected/28591d5d-928e-407f-b292-81ce32dc3393-kube-api-access-7549d\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:23.326481 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:23.326430    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"
Oct 30 22:30:23.327048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:23.326438    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj" event={"ID":"28591d5d-928e-407f-b292-81ce32dc3393","Type":"ContainerDied","Data":"1c175747b20e453835ceb8504ae3c86eaac0e48a08486ff4e3dd46c07f317967"}
Oct 30 22:30:23.327228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:23.327144    4282 scope.go:122] "RemoveContainer" containerID="2c8673c4c14241b05057ca361888516b4ff8cf8fbaa663b2dff2c5ed6dc77857"
Oct 30 22:30:23.359028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:23.359005    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"]
Oct 30 22:30:23.364360 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:23.364339    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889990-pgmvj"]
Oct 30 22:30:23.465671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:23.465633    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" path="/var/lib/kubelet/pods/19855a1c-4f0c-478c-acf4-8eb39097fc92/volumes"
Oct 30 22:30:23.466213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:23.466191    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28591d5d-928e-407f-b292-81ce32dc3393" path="/var/lib/kubelet/pods/28591d5d-928e-407f-b292-81ce32dc3393/volumes"
Oct 30 22:30:34.652508 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:34.652454    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:30:36.388825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:36.388791    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"]
Oct 30 22:30:36.389283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:36.389062    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy" containerID="cri-o://f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b" gracePeriod=30
Oct 30 22:30:38.942795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:38.942763    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:30:41.676823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.676799    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:30:41.801965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.801930    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-certs\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802022    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-data\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802070    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-kube-api-access-vf2qx\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-kube-api-access-vf2qx\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802289 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802194    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istiod-ca-cert\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802289 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802239    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-socket\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802276    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-certs" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:41.802405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802296    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-token\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802405 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802390    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-envoy\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802444    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-podinfo\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802594    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-socket" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:41.802740 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802712    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-credential-socket\") pod \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\" (UID: \"ba075b75-0bd2-45db-b1d7-a2b61edd6508\") "
Oct 30 22:30:41.802804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.802755    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istiod-ca-cert" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:41.803057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.803032    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-data" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:41.803632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.803612    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:41.804427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.804294    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:41.804427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.804345    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:41.804427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.804362    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:41.804735 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.804702    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-credential-socket" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:41.805945 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.805922    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-kube-api-access-vf2qx" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "kube-api-access-vf2qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:41.806196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.806175    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-podinfo" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:41.806697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.806679    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-token" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:41.806772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.806750    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-envoy" pod "ba075b75-0bd2-45db-b1d7-a2b61edd6508" (UID: "ba075b75-0bd2-45db-b1d7-a2b61edd6508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:41.906215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.906183    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2qx\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-kube-api-access-vf2qx\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:41.906215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.906215    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:41.906463 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.906225    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:41.906463 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.906233    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba075b75-0bd2-45db-b1d7-a2b61edd6508-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:41.906463 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:41.906241    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba075b75-0bd2-45db-b1d7-a2b61edd6508-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:42.523264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.523223    4282 generic.go:356] "Generic (PLEG): container finished" podID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerID="f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b" exitCode=0
Oct 30 22:30:42.523500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.523292    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"
Oct 30 22:30:42.523500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.523309    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" event={"ID":"ba075b75-0bd2-45db-b1d7-a2b61edd6508","Type":"ContainerDied","Data":"f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b"}
Oct 30 22:30:42.523500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.523376    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl" event={"ID":"ba075b75-0bd2-45db-b1d7-a2b61edd6508","Type":"ContainerDied","Data":"798a295c4afb28bef4211d8527822e454262fad21fc9ca32ff47ffa446f1f8d7"}
Oct 30 22:30:42.523500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.523409    4282 scope.go:122] "RemoveContainer" containerID="f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b"
Oct 30 22:30:42.537516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.537498    4282 scope.go:122] "RemoveContainer" containerID="f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b"
Oct 30 22:30:42.537844 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:42.537825    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b\": container with ID starting with f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b not found: ID does not exist" containerID="f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b"
Oct 30 22:30:42.537917 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.537857    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b"} err="failed to get container status \"f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b\": rpc error: code = NotFound desc = could not find container \"f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b\": container with ID starting with f37dc957d6b29ecedd51da05f16df3e99ce77671ec3b1a27d39493049c0a3e7b not found: ID does not exist"
Oct 30 22:30:42.553964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.553939    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"]
Oct 30 22:30:42.581327 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.581307    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"]
Oct 30 22:30:42.606836 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:42.606810    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-rwlnl"]
Oct 30 22:30:43.468550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:43.468517    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" path="/var/lib/kubelet/pods/ba075b75-0bd2-45db-b1d7-a2b61edd6508/volumes"
Oct 30 22:30:57.577217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.577173    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m" podUID="41e3b9df-e8c1-45f9-8e01-baa67142e7ca" containerName="main" containerID="cri-o://98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c" gracePeriod=45
Oct 30 22:30:57.856896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.856872    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:30:57.947322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.947292    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-dshm\") pod \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") "
Oct 30 22:30:57.947532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.947339    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-kube-api-access-m5z6k\" (UniqueName: \"kubernetes.io/projected/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-kube-api-access-m5z6k\") pod \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") "
Oct 30 22:30:57.947532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.947379    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tls-certs\") pod \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") "
Oct 30 22:30:57.947626 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.947574    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-model-cache\") pod \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") "
Oct 30 22:30:57.947626 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.947597    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tmp-dir\") pod \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") "
Oct 30 22:30:57.947714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.947690    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-home\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-home\") pod \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\" (UID: \"41e3b9df-e8c1-45f9-8e01-baa67142e7ca\") "
Oct 30 22:30:57.947883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.947859    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-model-cache" pod "41e3b9df-e8c1-45f9-8e01-baa67142e7ca" (UID: "41e3b9df-e8c1-45f9-8e01-baa67142e7ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:57.947978 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.947939    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tmp-dir" pod "41e3b9df-e8c1-45f9-8e01-baa67142e7ca" (UID: "41e3b9df-e8c1-45f9-8e01-baa67142e7ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:57.948142 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.948119    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-home" pod "41e3b9df-e8c1-45f9-8e01-baa67142e7ca" (UID: "41e3b9df-e8c1-45f9-8e01-baa67142e7ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:57.948420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.948405    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:57.948493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.948423    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:57.948493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.948432    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:57.950394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.950369    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tls-certs" pod "41e3b9df-e8c1-45f9-8e01-baa67142e7ca" (UID: "41e3b9df-e8c1-45f9-8e01-baa67142e7ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:57.950510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.950407    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-dshm" pod "41e3b9df-e8c1-45f9-8e01-baa67142e7ca" (UID: "41e3b9df-e8c1-45f9-8e01-baa67142e7ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:57.950510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:57.950455    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-kube-api-access-m5z6k" pod "41e3b9df-e8c1-45f9-8e01-baa67142e7ca" (UID: "41e3b9df-e8c1-45f9-8e01-baa67142e7ca"). InnerVolumeSpecName "kube-api-access-m5z6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:58.053374 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.053123    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:58.056286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.054455    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5z6k\" (UniqueName: \"kubernetes.io/projected/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-kube-api-access-m5z6k\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:58.056286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.056257    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e3b9df-e8c1-45f9-8e01-baa67142e7ca-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:58.653429 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.653398    4282 generic.go:356] "Generic (PLEG): container finished" podID="41e3b9df-e8c1-45f9-8e01-baa67142e7ca" containerID="98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c" exitCode=0
Oct 30 22:30:58.653862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.653444    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"
Oct 30 22:30:58.653862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.653448    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m" event={"ID":"41e3b9df-e8c1-45f9-8e01-baa67142e7ca","Type":"ContainerDied","Data":"98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c"}
Oct 30 22:30:58.653862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.653600    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m" event={"ID":"41e3b9df-e8c1-45f9-8e01-baa67142e7ca","Type":"ContainerDied","Data":"2bef9fb9edf37f449fac30df079a527887f50355d1aa8fc5190372ff0d095e0e"}
Oct 30 22:30:58.653862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.653626    4282 scope.go:122] "RemoveContainer" containerID="98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c"
Oct 30 22:30:58.667006 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.666986    4282 scope.go:122] "RemoveContainer" containerID="98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c"
Oct 30 22:30:58.667261 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:30:58.667244    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c\": container with ID starting with 98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c not found: ID does not exist" containerID="98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c"
Oct 30 22:30:58.667311 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.667269    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c"} err="failed to get container status \"98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c\": rpc error: code = NotFound desc = could not find container \"98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c\": container with ID starting with 98c1d4cc84caec22f0cf4f1ddca7f3082ab39d412ce9e6bcdffeb15bf1cafe1c not found: ID does not exist"
Oct 30 22:30:58.687937 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.687912    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"]
Oct 30 22:30:58.697584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:58.697560    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-28d39b/test-model-28d39b-kserve-587775484c-vg46m"]
Oct 30 22:30:59.473754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:30:59.473721    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e3b9df-e8c1-45f9-8e01-baa67142e7ca" path="/var/lib/kubelet/pods/41e3b9df-e8c1-45f9-8e01-baa67142e7ca/volumes"
Oct 30 22:31:00.978748 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:00.978712    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"]
Oct 30 22:31:04.818220 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:31:04.818169    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:31:15.365328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:15.365296    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"]
Oct 30 22:31:16.005322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.005279    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8" podUID="a002efa8-1326-4799-b1a9-c479bec25a4d" containerName="main" containerID="cri-o://c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f" gracePeriod=45
Oct 30 22:31:16.299529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.299497    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:31:16.375583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.375542    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-home\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-home\") pod \"a002efa8-1326-4799-b1a9-c479bec25a4d\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") "
Oct 30 22:31:16.376094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.375642    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a002efa8-1326-4799-b1a9-c479bec25a4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a002efa8-1326-4799-b1a9-c479bec25a4d-tls-certs\") pod \"a002efa8-1326-4799-b1a9-c479bec25a4d\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") "
Oct 30 22:31:16.376094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.375791    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a002efa8-1326-4799-b1a9-c479bec25a4d-kube-api-access-cb55g\" (UniqueName: \"kubernetes.io/projected/a002efa8-1326-4799-b1a9-c479bec25a4d-kube-api-access-cb55g\") pod \"a002efa8-1326-4799-b1a9-c479bec25a4d\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") "
Oct 30 22:31:16.376094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.375842    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-home" pod "a002efa8-1326-4799-b1a9-c479bec25a4d" (UID: "a002efa8-1326-4799-b1a9-c479bec25a4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.376094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.375868    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-tmp-dir\") pod \"a002efa8-1326-4799-b1a9-c479bec25a4d\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") "
Oct 30 22:31:16.376094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.376018    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-dshm\") pod \"a002efa8-1326-4799-b1a9-c479bec25a4d\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") "
Oct 30 22:31:16.376406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.376114    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-model-cache\") pod \"a002efa8-1326-4799-b1a9-c479bec25a4d\" (UID: \"a002efa8-1326-4799-b1a9-c479bec25a4d\") "
Oct 30 22:31:16.376406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.376122    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-tmp-dir" pod "a002efa8-1326-4799-b1a9-c479bec25a4d" (UID: "a002efa8-1326-4799-b1a9-c479bec25a4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.376556 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.376489    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-model-cache" pod "a002efa8-1326-4799-b1a9-c479bec25a4d" (UID: "a002efa8-1326-4799-b1a9-c479bec25a4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.377166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.377139    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.377290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.377171    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.377290 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.377188    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.378535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.378506    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-dshm" pod "a002efa8-1326-4799-b1a9-c479bec25a4d" (UID: "a002efa8-1326-4799-b1a9-c479bec25a4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:16.378695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.378667    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a002efa8-1326-4799-b1a9-c479bec25a4d-tls-certs" pod "a002efa8-1326-4799-b1a9-c479bec25a4d" (UID: "a002efa8-1326-4799-b1a9-c479bec25a4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:16.379026 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.379001    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a002efa8-1326-4799-b1a9-c479bec25a4d-kube-api-access-cb55g" pod "a002efa8-1326-4799-b1a9-c479bec25a4d" (UID: "a002efa8-1326-4799-b1a9-c479bec25a4d"). InnerVolumeSpecName "kube-api-access-cb55g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:16.478562 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.478534    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a002efa8-1326-4799-b1a9-c479bec25a4d-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.478562 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.478559    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb55g\" (UniqueName: \"kubernetes.io/projected/a002efa8-1326-4799-b1a9-c479bec25a4d-kube-api-access-cb55g\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.478761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.478569    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a002efa8-1326-4799-b1a9-c479bec25a4d-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:16.892035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.891997    4282 generic.go:356] "Generic (PLEG): container finished" podID="a002efa8-1326-4799-b1a9-c479bec25a4d" containerID="c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f" exitCode=0
Oct 30 22:31:16.892247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.892047    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"
Oct 30 22:31:16.892247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.892093    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8" event={"ID":"a002efa8-1326-4799-b1a9-c479bec25a4d","Type":"ContainerDied","Data":"c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f"}
Oct 30 22:31:16.892247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.892139    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8" event={"ID":"a002efa8-1326-4799-b1a9-c479bec25a4d","Type":"ContainerDied","Data":"b315b8376d155ecfe0a04dcc7496c2bd902c7347dae9559dc1bf0d242243416b"}
Oct 30 22:31:16.892247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.892159    4282 scope.go:122] "RemoveContainer" containerID="c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f"
Oct 30 22:31:16.905548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.905522    4282 scope.go:122] "RemoveContainer" containerID="c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f"
Oct 30 22:31:16.905841 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:31:16.905822    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f\": container with ID starting with c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f not found: ID does not exist" containerID="c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f"
Oct 30 22:31:16.905910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.905849    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f"} err="failed to get container status \"c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f\": rpc error: code = NotFound desc = could not find container \"c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f\": container with ID starting with c0ec8e15e80851cfa20ccbb7087d22f997e0843907889b1c59c2e1cbbbb8615f not found: ID does not exist"
Oct 30 22:31:16.925775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.925751    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"]
Oct 30 22:31:16.929794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:16.929774    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c92aec/test-model-c92aec-kserve-5765d74789-6rxm8"]
Oct 30 22:31:17.466042 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:17.466011    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a002efa8-1326-4799-b1a9-c479bec25a4d" path="/var/lib/kubelet/pods/a002efa8-1326-4799-b1a9-c479bec25a4d/volumes"
Oct 30 22:31:30.389204 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.389152    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf" podUID="bc3156f5-7370-486a-89fb-65087b1f969e" containerName="main" containerID="cri-o://9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53" gracePeriod=45
Oct 30 22:31:30.948573 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.948544    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:31:30.991438 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.991390    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-home\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-home\") pod \"bc3156f5-7370-486a-89fb-65087b1f969e\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") "
Oct 30 22:31:30.991738 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.991716    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-model-cache\") pod \"bc3156f5-7370-486a-89fb-65087b1f969e\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") "
Oct 30 22:31:30.992014 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.991994    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-tmp-dir\") pod \"bc3156f5-7370-486a-89fb-65087b1f969e\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") "
Oct 30 22:31:30.992255 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.992236    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-dshm\") pod \"bc3156f5-7370-486a-89fb-65087b1f969e\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") "
Oct 30 22:31:30.992544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.992523    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3156f5-7370-486a-89fb-65087b1f969e-kube-api-access-x2n69\" (UniqueName: \"kubernetes.io/projected/bc3156f5-7370-486a-89fb-65087b1f969e-kube-api-access-x2n69\") pod \"bc3156f5-7370-486a-89fb-65087b1f969e\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") "
Oct 30 22:31:30.993105 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.993085    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3156f5-7370-486a-89fb-65087b1f969e-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3156f5-7370-486a-89fb-65087b1f969e-tls-certs\") pod \"bc3156f5-7370-486a-89fb-65087b1f969e\" (UID: \"bc3156f5-7370-486a-89fb-65087b1f969e\") "
Oct 30 22:31:30.993288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.991795    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-home" pod "bc3156f5-7370-486a-89fb-65087b1f969e" (UID: "bc3156f5-7370-486a-89fb-65087b1f969e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.993392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.992043    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-model-cache" pod "bc3156f5-7370-486a-89fb-65087b1f969e" (UID: "bc3156f5-7370-486a-89fb-65087b1f969e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.993392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.992282    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-tmp-dir" pod "bc3156f5-7370-486a-89fb-65087b1f969e" (UID: "bc3156f5-7370-486a-89fb-65087b1f969e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.994426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.994405    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.994632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.994616    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.994746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.994732    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:30.995500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.995425    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-dshm" pod "bc3156f5-7370-486a-89fb-65087b1f969e" (UID: "bc3156f5-7370-486a-89fb-65087b1f969e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:30.995715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.995688    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3156f5-7370-486a-89fb-65087b1f969e-kube-api-access-x2n69" pod "bc3156f5-7370-486a-89fb-65087b1f969e" (UID: "bc3156f5-7370-486a-89fb-65087b1f969e"). InnerVolumeSpecName "kube-api-access-x2n69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:30.995959 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:30.995938    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3156f5-7370-486a-89fb-65087b1f969e-tls-certs" pod "bc3156f5-7370-486a-89fb-65087b1f969e" (UID: "bc3156f5-7370-486a-89fb-65087b1f969e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:31.069256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.069226    4282 generic.go:356] "Generic (PLEG): container finished" podID="bc3156f5-7370-486a-89fb-65087b1f969e" containerID="9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53" exitCode=0
Oct 30 22:31:31.069490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.069280    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"
Oct 30 22:31:31.069490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.069318    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf" event={"ID":"bc3156f5-7370-486a-89fb-65087b1f969e","Type":"ContainerDied","Data":"9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53"}
Oct 30 22:31:31.069490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.069363    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf" event={"ID":"bc3156f5-7370-486a-89fb-65087b1f969e","Type":"ContainerDied","Data":"1e3c1bd14c33f421867db5514578d79dcb2511f7115274dcc81ca2340bf89150"}
Oct 30 22:31:31.069490 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.069379    4282 scope.go:122] "RemoveContainer" containerID="9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53"
Oct 30 22:31:31.081713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.081588    4282 scope.go:122] "RemoveContainer" containerID="9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53"
Oct 30 22:31:31.081912 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:31:31.081890    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53\": container with ID starting with 9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53 not found: ID does not exist" containerID="9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53"
Oct 30 22:31:31.081988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.081919    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53"} err="failed to get container status \"9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53\": rpc error: code = NotFound desc = could not find container \"9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53\": container with ID starting with 9c0e24b26bf59174dbfc4ff1225feb2c23f3852b9242837a517835d36c381e53 not found: ID does not exist"
Oct 30 22:31:31.096460 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.096425    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3156f5-7370-486a-89fb-65087b1f969e-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:31.096611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.096503    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2n69\" (UniqueName: \"kubernetes.io/projected/bc3156f5-7370-486a-89fb-65087b1f969e-kube-api-access-x2n69\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:31.096611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.096523    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3156f5-7370-486a-89fb-65087b1f969e-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:31.106018 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.105995    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"]
Oct 30 22:31:31.113227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.113201    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-120cb1/test-model-120cb1-kserve-686f649b4f-9w8nf"]
Oct 30 22:31:31.403025 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.402948    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"]
Oct 30 22:31:31.403454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.403210    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy" containerID="cri-o://7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed" gracePeriod=30
Oct 30 22:31:31.466535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:31.466504    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3156f5-7370-486a-89fb-65087b1f969e" path="/var/lib/kubelet/pods/bc3156f5-7370-486a-89fb-65087b1f969e/volumes"
Oct 30 22:31:32.095476 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:32.095432    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:31:32.948529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:32.948450    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"]
Oct 30 22:31:32.949404 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:32.949348    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy" containerID="cri-o://c5383f9420afadfc42e2da6d068fe67c30988dfdbd90391ed4ec548c213bc6e3" gracePeriod=30
Oct 30 22:31:32.966996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:32.966960    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"]
Oct 30 22:31:32.967449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:32.967414    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy" containerID="cri-o://494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d" gracePeriod=30
Oct 30 22:31:34.672508 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:31:34.672452    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:31:36.502370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.502326    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:31:36.502788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.502391    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:31:36.687452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.687387    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:31:36.801834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.801792    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-kube-api-access-j2qkr\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-kube-api-access-j2qkr\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.801852    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-socket\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.801937    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-istio-token\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-istio-token\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.801976    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-data\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802028 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802019    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a223810-9530-490c-8cfe-82e27f55b98b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a223810-9530-490c-8cfe-82e27f55b98b-istiod-ca-cert\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802064    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-certs\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802127    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a223810-9530-490c-8cfe-82e27f55b98b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a223810-9530-490c-8cfe-82e27f55b98b-istio-podinfo\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802233    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-socket" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:36.802392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802306    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-envoy\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802392 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802353    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-credential-socket\") pod \"3a223810-9530-490c-8cfe-82e27f55b98b\" (UID: \"3a223810-9530-490c-8cfe-82e27f55b98b\") "
Oct 30 22:31:36.802537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802490    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-certs" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:36.802669 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802609    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a223810-9530-490c-8cfe-82e27f55b98b-istiod-ca-cert" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:36.802804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802682    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-data" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:36.802946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.802829    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-credential-socket" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:36.803269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.803022    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:36.803269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.803043    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:36.803269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.803055    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:36.803269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.803063    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a223810-9530-490c-8cfe-82e27f55b98b-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:36.803269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.803072    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:36.846517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.826639    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-kube-api-access-j2qkr" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "kube-api-access-j2qkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:36.850264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.850111    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-envoy" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:36.850264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.850182    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-istio-token" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:36.850264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.850209    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a223810-9530-490c-8cfe-82e27f55b98b-istio-podinfo" pod "3a223810-9530-490c-8cfe-82e27f55b98b" (UID: "3a223810-9530-490c-8cfe-82e27f55b98b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:36.905100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.905066    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:36.905100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.905096    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a223810-9530-490c-8cfe-82e27f55b98b-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:36.905100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.905106    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a223810-9530-490c-8cfe-82e27f55b98b-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:36.905356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:36.905115    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2qkr\" (UniqueName: \"kubernetes.io/projected/3a223810-9530-490c-8cfe-82e27f55b98b-kube-api-access-j2qkr\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:37.311795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.311440    4282 generic.go:356] "Generic (PLEG): container finished" podID="3a223810-9530-490c-8cfe-82e27f55b98b" containerID="7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed" exitCode=0
Oct 30 22:31:37.311795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.311543    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" event={"ID":"3a223810-9530-490c-8cfe-82e27f55b98b","Type":"ContainerDied","Data":"7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed"}
Oct 30 22:31:37.311795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.311591    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv" event={"ID":"3a223810-9530-490c-8cfe-82e27f55b98b","Type":"ContainerDied","Data":"234265c1b0a3fa8c6d298d814fc297c199dd1e600477e97d601aa44a0e3721c2"}
Oct 30 22:31:37.311795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.311622    4282 scope.go:122] "RemoveContainer" containerID="7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed"
Oct 30 22:31:37.311795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.311792    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"
Oct 30 22:31:37.360586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.360546    4282 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:31:37.360820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.360612    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:31:37.408620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.408557    4282 scope.go:122] "RemoveContainer" containerID="7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed"
Oct 30 22:31:37.411499 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:31:37.410215    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed\": container with ID starting with 7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed not found: ID does not exist" containerID="7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed"
Oct 30 22:31:37.411499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.410255    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed"} err="failed to get container status \"7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed\": rpc error: code = NotFound desc = could not find container \"7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed\": container with ID starting with 7990bbdb6a41259bf45b0cf2411a2a0ec22183f783c290a3e01da4859d5cb7ed not found: ID does not exist"
Oct 30 22:31:37.417480 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.417381    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"]
Oct 30 22:31:37.424268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.424240    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-bj2pv"]
Oct 30 22:31:37.466652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:37.466616    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" path="/var/lib/kubelet/pods/3a223810-9530-490c-8cfe-82e27f55b98b/volumes"
Oct 30 22:31:38.267370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.267345    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:31:38.328570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.328542    4282 generic.go:356] "Generic (PLEG): container finished" podID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerID="c5383f9420afadfc42e2da6d068fe67c30988dfdbd90391ed4ec548c213bc6e3" exitCode=0
Oct 30 22:31:38.328729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.328618    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" event={"ID":"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e","Type":"ContainerDied","Data":"c5383f9420afadfc42e2da6d068fe67c30988dfdbd90391ed4ec548c213bc6e3"}
Oct 30 22:31:38.330296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.330269    4282 generic.go:356] "Generic (PLEG): container finished" podID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerID="494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d" exitCode=0
Oct 30 22:31:38.330452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.330403    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"
Oct 30 22:31:38.330452 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.330412    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" event={"ID":"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d","Type":"ContainerDied","Data":"494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d"}
Oct 30 22:31:38.330652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.330488    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m" event={"ID":"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d","Type":"ContainerDied","Data":"2b131d1e4e9ef0f1238f78df7d9be99a9e8b480283814d63f21cbdd12c11b70a"}
Oct 30 22:31:38.330652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.330525    4282 scope.go:122] "RemoveContainer" containerID="494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d"
Oct 30 22:31:38.337003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.336980    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-kube-api-access-pplkw\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-kube-api-access-pplkw\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.337148 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337086    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-data\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.337226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337170    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-socket\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.337293 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337243    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-token\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-token\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.337293 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337289    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-certs\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.337413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337341    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istiod-ca-cert\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.337413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337374    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-data" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.337550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337504    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-socket" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.337550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337510    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-podinfo\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.337684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337637    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-credential-socket\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.338660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337761    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-envoy\") pod \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\" (UID: \"733b9d66-0f71-4d1a-9f57-8bd277cc7e2d\") "
Oct 30 22:31:38.338660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.337971    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-certs" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.338660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.338389    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-credential-socket" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.338660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.338563    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istiod-ca-cert" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:38.339005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.338980    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.339074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.339013    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.339074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.339029    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.339074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.339043    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.339074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.339058    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.339980 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.339952    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-kube-api-access-pplkw" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "kube-api-access-pplkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:38.340114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.340093    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-podinfo" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:38.340178 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.340137    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-token" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:38.341967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.341932    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-envoy" pod "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" (UID: "733b9d66-0f71-4d1a-9f57-8bd277cc7e2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.344000 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.343976    4282 scope.go:122] "RemoveContainer" containerID="494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d"
Oct 30 22:31:38.344368 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:31:38.344345    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d\": container with ID starting with 494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d not found: ID does not exist" containerID="494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d"
Oct 30 22:31:38.344515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.344377    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d"} err="failed to get container status \"494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d\": rpc error: code = NotFound desc = could not find container \"494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d\": container with ID starting with 494053c53e0fa96d8a5e8eded2dfa36042e2ce5701952d754d9446f4cfc2764d not found: ID does not exist"
Oct 30 22:31:38.444741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.443820    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.444741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.443852    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pplkw\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-kube-api-access-pplkw\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.444741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.443870    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.444741 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.443882    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.657459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.657437    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:31:38.670598 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.670565    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"]
Oct 30 22:31:38.674594 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.674572    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-h8s7m"]
Oct 30 22:31:38.749659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.749624    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-data\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.749846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.749739    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-envoy\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.749846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.749777    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-certs\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.749846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.749838    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-credential-socket\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.750021 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.749896    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-kube-api-access-v4qnh\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-kube-api-access-v4qnh\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.750021 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.749907    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-data" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.750021 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.750003    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-socket\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.750906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.750158    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-podinfo\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.750906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.750219    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-token\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-token\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.750906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.750236    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-credential-socket" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.750906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.750276    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istiod-ca-cert\") pod \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\" (UID: \"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e\") "
Oct 30 22:31:38.750906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.750449    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-socket" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.751526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.751005    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istiod-ca-cert" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:38.752070 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.752047    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.752229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.752076    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.752229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.752092    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.752229 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.752106    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.752535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.752388    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-certs" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.752912 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.752879    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-envoy" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:38.753848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.753802    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-token" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:38.754862 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.754836    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-podinfo" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:38.754967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.754946    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-kube-api-access-v4qnh" pod "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" (UID: "963db1f0-ee51-4b7d-a0fc-c3913e5cce2e"). InnerVolumeSpecName "kube-api-access-v4qnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:38.854013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.853931    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4qnh\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-kube-api-access-v4qnh\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.854013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.853962    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.854013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.853976    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.854013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.853990    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:38.854013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:38.854002    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:39.340866 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:39.340832    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w" event={"ID":"963db1f0-ee51-4b7d-a0fc-c3913e5cce2e","Type":"ContainerDied","Data":"427a437074f3b008f15deb90a2ea5aca3c21a261b97b00e32dbe01f2a9748cb0"}
Oct 30 22:31:39.341330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:39.340900    4282 scope.go:122] "RemoveContainer" containerID="c5383f9420afadfc42e2da6d068fe67c30988dfdbd90391ed4ec548c213bc6e3"
Oct 30 22:31:39.341330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:39.340845    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"
Oct 30 22:31:39.366181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:39.366149    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"]
Oct 30 22:31:39.371848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:39.371828    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-cbg2w"]
Oct 30 22:31:39.466756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:39.466725    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" path="/var/lib/kubelet/pods/733b9d66-0f71-4d1a-9f57-8bd277cc7e2d/volumes"
Oct 30 22:31:39.469497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:39.467872    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" path="/var/lib/kubelet/pods/963db1f0-ee51-4b7d-a0fc-c3913e5cce2e/volumes"
Oct 30 22:31:42.009890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:42.009855    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"]
Oct 30 22:31:42.023371 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:42.023340    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"]
Oct 30 22:31:43.771157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:43.771125    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"]
Oct 30 22:31:43.787287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:43.787247    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"]
Oct 30 22:31:47.010553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.010500    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8" podUID="a715bc3b-bebf-4a86-b0dc-ffd695a2854b" containerName="payload-processing" containerID="cri-o://666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06" gracePeriod=25
Oct 30 22:31:47.024113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.024052    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh" podUID="31d7feb5-8074-4993-ae62-5352da049922" containerName="payload-pre-processing" containerID="cri-o://015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1" gracePeriod=25
Oct 30 22:31:47.306096 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.306072    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:31:47.470399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.470371    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-kube-api-access-n4cvj\" (UniqueName: \"kubernetes.io/projected/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-kube-api-access-n4cvj\") pod \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\" (UID: \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\") "
Oct 30 22:31:47.470638 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.470544    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-plugins-config-volume\") pod \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\" (UID: \"a715bc3b-bebf-4a86-b0dc-ffd695a2854b\") "
Oct 30 22:31:47.470897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.470872    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-plugins-config-volume" pod "a715bc3b-bebf-4a86-b0dc-ffd695a2854b" (UID: "a715bc3b-bebf-4a86-b0dc-ffd695a2854b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:47.471126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.471098    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:47.472592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.472572    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-kube-api-access-n4cvj" pod "a715bc3b-bebf-4a86-b0dc-ffd695a2854b" (UID: "a715bc3b-bebf-4a86-b0dc-ffd695a2854b"). InnerVolumeSpecName "kube-api-access-n4cvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:47.491257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.491233    4282 generic.go:356] "Generic (PLEG): container finished" podID="a715bc3b-bebf-4a86-b0dc-ffd695a2854b" containerID="666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06" exitCode=0
Oct 30 22:31:47.491379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.491284    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"
Oct 30 22:31:47.491379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.491310    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8" event={"ID":"a715bc3b-bebf-4a86-b0dc-ffd695a2854b","Type":"ContainerDied","Data":"666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06"}
Oct 30 22:31:47.491379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.491361    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8" event={"ID":"a715bc3b-bebf-4a86-b0dc-ffd695a2854b","Type":"ContainerDied","Data":"8e15e61e5c2462b0ee342f84d1ff01fd081eabd18bcd2fe7be07b8f5e70336f1"}
Oct 30 22:31:47.491580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.491385    4282 scope.go:122] "RemoveContainer" containerID="666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06"
Oct 30 22:31:47.502417 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.502396    4282 scope.go:122] "RemoveContainer" containerID="666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06"
Oct 30 22:31:47.502729 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:31:47.502709    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06\": container with ID starting with 666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06 not found: ID does not exist" containerID="666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06"
Oct 30 22:31:47.502787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.502738    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06"} err="failed to get container status \"666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06\": rpc error: code = NotFound desc = could not find container \"666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06\": container with ID starting with 666ca9dbbd632ab35426dece94c5f85da9cbc6769a4101257e7604f6a54b8f06 not found: ID does not exist"
Oct 30 22:31:47.516141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.516087    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"]
Oct 30 22:31:47.520315 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.520292    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-58d5fa-7b859657fc-ppfb8"]
Oct 30 22:31:47.594499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:47.586512    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cvj\" (UniqueName: \"kubernetes.io/projected/a715bc3b-bebf-4a86-b0dc-ffd695a2854b-kube-api-access-n4cvj\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:48.772149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:48.772094    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592" podUID="fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" containerName="payload-processing" containerID="cri-o://48933d7f3cad28e1acf8fea2aeeaa0314669484c87bb6ef224162a510135fc0e" gracePeriod=25
Oct 30 22:31:48.788509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:48.788476    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx" podUID="303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" containerName="payload-pre-processing" containerID="cri-o://a0356b309a9659226cacda5d202852280ce27b66e719d5cc060f657e3253cd8c" gracePeriod=25
Oct 30 22:31:49.552526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:49.544883    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a715bc3b-bebf-4a86-b0dc-ffd695a2854b" path="/var/lib/kubelet/pods/a715bc3b-bebf-4a86-b0dc-ffd695a2854b/volumes"
Oct 30 22:31:50.351586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.351549    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"]
Oct 30 22:31:50.352324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352307    4282 cpu_manager.go:414] "containerMap: removing container" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy"
Oct 30 22:31:50.352324 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352324    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352335    4282 cpu_manager.go:414] "containerMap: removing container" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352341    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352347    4282 cpu_manager.go:414] "containerMap: removing container" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352352    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352359    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352364    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352371    4282 cpu_manager.go:414] "containerMap: removing container" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352376    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352391    4282 cpu_manager.go:414] "containerMap: removing container" podUID="bc3156f5-7370-486a-89fb-65087b1f969e" containerName="main"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352396    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3156f5-7370-486a-89fb-65087b1f969e" containerName="main"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352404    4282 cpu_manager.go:414] "containerMap: removing container" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352408    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352424    4282 cpu_manager.go:414] "containerMap: removing container" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352429    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352438    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352443    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352452    4282 cpu_manager.go:414] "containerMap: removing container" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352456    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352462    4282 cpu_manager.go:414] "containerMap: removing container" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352481    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352490    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352494    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352500    4282 cpu_manager.go:414] "containerMap: removing container" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352504    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352512    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352516    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352523    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a715bc3b-bebf-4a86-b0dc-ffd695a2854b" containerName="payload-processing"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352527    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a715bc3b-bebf-4a86-b0dc-ffd695a2854b" containerName="payload-processing"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352533    4282 cpu_manager.go:414] "containerMap: removing container" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:31:50.352519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352538    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352544    4282 cpu_manager.go:414] "containerMap: removing container" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352548    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352556    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352559    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352565    4282 cpu_manager.go:414] "containerMap: removing container" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352569    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352577    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352582    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352590    4282 cpu_manager.go:414] "containerMap: removing container" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352594    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352601    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352606    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352613    4282 cpu_manager.go:414] "containerMap: removing container" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352617    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352625    4282 cpu_manager.go:414] "containerMap: removing container" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352629    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352636    4282 cpu_manager.go:414] "containerMap: removing container" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352640    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352645    4282 cpu_manager.go:414] "containerMap: removing container" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352649    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352656    4282 cpu_manager.go:414] "containerMap: removing container" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352661    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352670    4282 cpu_manager.go:414] "containerMap: removing container" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352675    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352682    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352686    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352706    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352711    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352717    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a002efa8-1326-4799-b1a9-c479bec25a4d" containerName="main"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352721    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a002efa8-1326-4799-b1a9-c479bec25a4d" containerName="main"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352727    4282 cpu_manager.go:414] "containerMap: removing container" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352730    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352739    4282 cpu_manager.go:414] "containerMap: removing container" podUID="41e3b9df-e8c1-45f9-8e01-baa67142e7ca" containerName="main"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352743    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e3b9df-e8c1-45f9-8e01-baa67142e7ca" containerName="main"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352848    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3156f5-7370-486a-89fb-65087b1f969e" containerName="main"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352857    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352864    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352869    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352873    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352879    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352883    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352889    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352894    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352899    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a715bc3b-bebf-4a86-b0dc-ffd695a2854b" containerName="payload-processing"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352905    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352910    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352916    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352921    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352928    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352932    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2ce866-ccbc-4279-915e-cb75f1338701" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352936    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352942    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352947    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352951    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352956    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="963db1f0-ee51-4b7d-a0fc-c3913e5cce2e" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352965    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba075b75-0bd2-45db-b1d7-a2b61edd6508" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352972    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a002efa8-1326-4799-b1a9-c479bec25a4d" containerName="main"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352977    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a223810-9530-490c-8cfe-82e27f55b98b" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352983    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e3b9df-e8c1-45f9-8e01-baa67142e7ca" containerName="main"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352989    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="733b9d66-0f71-4d1a-9f57-8bd277cc7e2d" containerName="istio-proxy"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.352995    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.353000    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:31:50.354145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.353005    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:31:50.357989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.357969    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.363286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.363249    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f60026-openshift-default-dockercfg-5slwr\""
Oct 30 22:31:50.370332 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.370307    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"]
Oct 30 22:31:50.453334 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.453297    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-certs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.453571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.453522    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-istio-token\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.453571 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.453564    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-envoy\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.453710 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.453658    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea8884-75a7-4608-b491-40cb633aca64-istio-podinfo\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.453786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.453768    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-data\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.453844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.453829    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.454044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.454028    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-credential-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.454103 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.454064    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcj8h\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-kube-api-access-hcj8h\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.454221 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.454194    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea8884-75a7-4608-b491-40cb633aca64-istiod-ca-cert\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555184 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555152    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-credential-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555184 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555192    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcj8h\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-kube-api-access-hcj8h\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555227    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea8884-75a7-4608-b491-40cb633aca64-istiod-ca-cert\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555251    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-certs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555337    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-istio-token\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555361    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-envoy\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555390    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea8884-75a7-4608-b491-40cb633aca64-istio-podinfo\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555505    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-data\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555549    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.555775 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555677    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-credential-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.556002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.555967    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-certs\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.556100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.556076    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-socket\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.556341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.556320    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea8884-75a7-4608-b491-40cb633aca64-istiod-ca-cert\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.556597 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.556577    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-data\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.557884 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.557862    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-envoy\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.558790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.558763    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea8884-75a7-4608-b491-40cb633aca64-istio-podinfo\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.573870 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.573843    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcj8h\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-kube-api-access-hcj8h\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.573996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.573974    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-istio-token\") pod \"e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") " pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.676754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.676715    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:50.992200 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:31:50.992170    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ea8884_75a7_4608_b491_40cb633aca64.slice/crio-f470cd053c555ffdb4d0bfa49a5af1037197fe43ced495ac7756bf7c93a60a26 WatchSource:0}: Error finding container f470cd053c555ffdb4d0bfa49a5af1037197fe43ced495ac7756bf7c93a60a26: Status 404 returned error can't find the container with id f470cd053c555ffdb4d0bfa49a5af1037197fe43ced495ac7756bf7c93a60a26
Oct 30 22:31:50.992333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.992208    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"]
Oct 30 22:31:50.994962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.994915    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:31:50.995075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.995007    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:31:50.995075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:50.995050    4282 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 22:31:51.154530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.146057    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"]
Oct 30 22:31:51.154530 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.146254    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" containerID="cri-o://efd2e4ae86e0db3cf23d0bfb8c41f65653965a3a13d0afdaf95d5ecdefb39c83" gracePeriod=30
Oct 30 22:31:51.168271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.168233    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"]
Oct 30 22:31:51.168569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.168523    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" containerID="cri-o://a4c047c8c6f94589b346f287701803612dbad1074e6910ed4496d5602eb6eaec" gracePeriod=30
Oct 30 22:31:51.175420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.175396    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"]
Oct 30 22:31:51.175616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.175597    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy" containerID="cri-o://e84bbca860e5e2fea50d5be7cb5d99e5b1716d58886e3598478e63b1da19edc3" gracePeriod=30
Oct 30 22:31:51.306570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.304990    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:31:51.306570 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.305050    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:31:51.402534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.399660    4282 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:31:51.402534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.399720    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:31:51.552390 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.552353    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" event={"ID":"c8ea8884-75a7-4608-b491-40cb633aca64","Type":"ContainerStarted","Data":"7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b"}
Oct 30 22:31:51.552633 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.552430    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" event={"ID":"c8ea8884-75a7-4608-b491-40cb633aca64","Type":"ContainerStarted","Data":"f470cd053c555ffdb4d0bfa49a5af1037197fe43ced495ac7756bf7c93a60a26"}
Oct 30 22:31:51.584013 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.583908    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" podStartSLOduration=1.583889841 podStartE2EDuration="1.583889841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:51.580677225 +0000 UTC m=+3130.238773449" watchObservedRunningTime="2026-10-30 22:31:51.583889841 +0000 UTC m=+3130.241986062"
Oct 30 22:31:51.680004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:51.678551    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:52.679170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:52.679131    4282 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:31:52.679745 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:52.679194    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 22:31:52.697551 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:52.697519    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592" podUID="fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:53.202493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:53.196614    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh" podUID="31d7feb5-8074-4993-ae62-5352da049922" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:53.678044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:53.677985    4282 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:31:53.678223 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:53.678060    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 22:31:53.747484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:53.747433    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx" podUID="303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:31:54.681847 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:54.681666    4282 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:31:54.681847 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:54.681726    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 22:31:55.677987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:55.677957    4282 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:31:55.678535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:55.678013    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 22:31:56.693784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.688696    4282 generic.go:356] "Generic (PLEG): container finished" podID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerID="e84bbca860e5e2fea50d5be7cb5d99e5b1716d58886e3598478e63b1da19edc3" exitCode=0
Oct 30 22:31:56.693784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.688992    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" event={"ID":"17a61cf4-e93a-4fad-a160-e6f385fcf88f","Type":"ContainerDied","Data":"e84bbca860e5e2fea50d5be7cb5d99e5b1716d58886e3598478e63b1da19edc3"}
Oct 30 22:31:56.693784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.691186    4282 generic.go:356] "Generic (PLEG): container finished" podID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerID="efd2e4ae86e0db3cf23d0bfb8c41f65653965a3a13d0afdaf95d5ecdefb39c83" exitCode=0
Oct 30 22:31:56.693784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.691382    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" event={"ID":"5fc6b7c6-166a-4d5f-8da3-ee7345189759","Type":"ContainerDied","Data":"efd2e4ae86e0db3cf23d0bfb8c41f65653965a3a13d0afdaf95d5ecdefb39c83"}
Oct 30 22:31:56.693784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.691431    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6" event={"ID":"5fc6b7c6-166a-4d5f-8da3-ee7345189759","Type":"ContainerDied","Data":"3568f359bf74ab7aff96ecc920943333dfd5d1a48a388619e9fb88830ec6143c"}
Oct 30 22:31:56.693784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.691491    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3568f359bf74ab7aff96ecc920943333dfd5d1a48a388619e9fb88830ec6143c"
Oct 30 22:31:56.694581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.693867    4282 generic.go:356] "Generic (PLEG): container finished" podID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerID="a4c047c8c6f94589b346f287701803612dbad1074e6910ed4496d5602eb6eaec" exitCode=0
Oct 30 22:31:56.694581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.694014    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" event={"ID":"d3e43d9d-1cf7-418f-a158-1a2d002dafcc","Type":"ContainerDied","Data":"a4c047c8c6f94589b346f287701803612dbad1074e6910ed4496d5602eb6eaec"}
Oct 30 22:31:56.694581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.694186    4282 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:31:56.694581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.694231    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 22:31:56.719782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.708762    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:31:56.725391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.725367    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:31:56.744854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.744815    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:31:56.813401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813365    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-token\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-token\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.813649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813421    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-data\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.813649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813452    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-envoy\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.813649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813516    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-certs\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.813649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813583    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-data\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.813825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813664    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istiod-ca-cert\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.813825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813695    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-podinfo\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.813825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813724    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istiod-ca-cert\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.813825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.813802    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-kube-api-access-6hrdb\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-kube-api-access-6hrdb\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.814194 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.814144    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-certs" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.814896 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.814856    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istiod-ca-cert" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:56.815167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.815111    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-data" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.815331 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.815297    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-data" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.816349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.815825    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istiod-ca-cert" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:56.816349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816127    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-credential-socket\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.816349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816172    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-kube-api-access-nzm7m\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-kube-api-access-nzm7m\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.816349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816229    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-socket\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.816349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816264    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-kube-api-access-72fz7\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-kube-api-access-72fz7\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.816349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816296    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-socket\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816369    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-token\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-token\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816406    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istiod-ca-cert\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816502    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-podinfo\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816544    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-credential-socket\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816599    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-data\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816671    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-podinfo\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816696    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-envoy\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816732    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-credential-socket\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816763    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-socket\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816785    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-envoy\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816849    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-token\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-token\") pod \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\" (UID: \"d3e43d9d-1cf7-418f-a158-1a2d002dafcc\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816908    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-certs\") pod \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\" (UID: \"17a61cf4-e93a-4fad-a160-e6f385fcf88f\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.816973    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-certs\") pod \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\" (UID: \"5fc6b7c6-166a-4d5f-8da3-ee7345189759\") "
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817630    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-token" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817848    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817872    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817872    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-socket" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817887    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817886    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istiod-ca-cert" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817903    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817917    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.817933    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.819168    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-kube-api-access-72fz7" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "kube-api-access-72fz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:56.819419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.819396    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-credential-socket" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.829264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829194    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-envoy" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.829264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829210    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-podinfo" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:56.829264 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829260    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-token" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:56.829602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829435    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-credential-socket" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.829602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829444    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-credential-socket" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.829723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829690    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-data" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.829723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829698    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-certs" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.829910 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829886    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-certs" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.829975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.829950    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-socket" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.830032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.830010    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-podinfo" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:56.830141 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.830121    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-socket" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.830637 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.830610    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-podinfo" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:56.831343 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.831313    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-kube-api-access-nzm7m" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "kube-api-access-nzm7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:56.831957 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.831897    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-envoy" pod "5fc6b7c6-166a-4d5f-8da3-ee7345189759" (UID: "5fc6b7c6-166a-4d5f-8da3-ee7345189759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.832480 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.832434    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-kube-api-access-6hrdb" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "kube-api-access-6hrdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:56.832827 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.832802    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-token" pod "d3e43d9d-1cf7-418f-a158-1a2d002dafcc" (UID: "d3e43d9d-1cf7-418f-a158-1a2d002dafcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:56.833428 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.833405    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-envoy" pod "17a61cf4-e93a-4fad-a160-e6f385fcf88f" (UID: "17a61cf4-e93a-4fad-a160-e6f385fcf88f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:56.919010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.918970    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919001    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919016    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919030    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919042    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919053    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919065    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919079    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919092    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919106    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919118    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919131    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919143    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919155    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919167    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fc6b7c6-166a-4d5f-8da3-ee7345189759-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919180    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hrdb\" (UniqueName: \"kubernetes.io/projected/17a61cf4-e93a-4fad-a160-e6f385fcf88f-kube-api-access-6hrdb\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919193    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919205    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzm7m\" (UniqueName: \"kubernetes.io/projected/d3e43d9d-1cf7-418f-a158-1a2d002dafcc-kube-api-access-nzm7m\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919218    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a61cf4-e93a-4fad-a160-e6f385fcf88f-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919230    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72fz7\" (UniqueName: \"kubernetes.io/projected/5fc6b7c6-166a-4d5f-8da3-ee7345189759-kube-api-access-72fz7\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:56.919286 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:56.919243    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fc6b7c6-166a-4d5f-8da3-ee7345189759-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:31:57.678083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.678047    4282 patch_prober.go:29] interesting pod/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body=
Oct 30 22:31:57.678272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.678109    4282 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused"
Oct 30 22:31:57.704362 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.704331    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"
Oct 30 22:31:57.704880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.704368    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m" event={"ID":"d3e43d9d-1cf7-418f-a158-1a2d002dafcc","Type":"ContainerDied","Data":"6732646f8e2f707dd641bd52c64b03cfbf07952f677ed9c4a579c1c7d8e135e3"}
Oct 30 22:31:57.704880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.704457    4282 scope.go:122] "RemoveContainer" containerID="a4c047c8c6f94589b346f287701803612dbad1074e6910ed4496d5602eb6eaec"
Oct 30 22:31:57.706491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.706448    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"
Oct 30 22:31:57.706611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.706512    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8" event={"ID":"17a61cf4-e93a-4fad-a160-e6f385fcf88f","Type":"ContainerDied","Data":"44cd1fd0b4305201e163cd79e1f2b4d7159a1d67a2909bcc07553e9457d25e25"}
Oct 30 22:31:57.706663 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.706456    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"
Oct 30 22:31:57.718565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.718546    4282 scope.go:122] "RemoveContainer" containerID="e84bbca860e5e2fea50d5be7cb5d99e5b1716d58886e3598478e63b1da19edc3"
Oct 30 22:31:57.736499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.736459    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"]
Oct 30 22:31:57.743273 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.743247    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-sjqq8"]
Oct 30 22:31:57.747255 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.747195    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"]
Oct 30 22:31:57.750714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.750694    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-9plh6"]
Oct 30 22:31:57.760377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.760356    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"]
Oct 30 22:31:57.767523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:57.767505    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-8zs8m"]
Oct 30 22:31:58.682232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:58.682204    4282 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:58.682742 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:58.682724    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:58.683627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:58.683608    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:31:59.434979 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:59.434944    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"]
Oct 30 22:31:59.435454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:59.435205    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy" containerID="cri-o://0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf" gracePeriod=30
Oct 30 22:31:59.470677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:59.470651    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" path="/var/lib/kubelet/pods/17a61cf4-e93a-4fad-a160-e6f385fcf88f/volumes"
Oct 30 22:31:59.471305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:59.471285    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" path="/var/lib/kubelet/pods/5fc6b7c6-166a-4d5f-8da3-ee7345189759/volumes"
Oct 30 22:31:59.471939 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:31:59.471922    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" path="/var/lib/kubelet/pods/d3e43d9d-1cf7-418f-a158-1a2d002dafcc/volumes"
Oct 30 22:32:01.805488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:01.805437    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"]
Oct 30 22:32:01.812732 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:01.812706    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"]
Oct 30 22:32:02.697370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:02.697331    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592" podUID="fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:03.178826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:03.178790    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh" podUID="31d7feb5-8074-4993-ae62-5352da049922" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:03.776600 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:03.776561    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx" podUID="303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:04.663177 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:04.663117    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:32:04.735199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.735176    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:32:04.806851 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.806811    4282 generic.go:356] "Generic (PLEG): container finished" podID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerID="0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf" exitCode=0
Oct 30 22:32:04.807071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.806862    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" event={"ID":"de4440c1-907a-4752-9a69-197dfcb5ead4","Type":"ContainerDied","Data":"0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf"}
Oct 30 22:32:04.807071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.806956    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v" event={"ID":"de4440c1-907a-4752-9a69-197dfcb5ead4","Type":"ContainerDied","Data":"56fa78b05577860567b5962291ad8ed29eff88cc14d5e017917a70065f32c268"}
Oct 30 22:32:04.807071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.807001    4282 scope.go:122] "RemoveContainer" containerID="0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf"
Oct 30 22:32:04.807274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.807073    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"
Oct 30 22:32:04.818611 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.818585    4282 scope.go:122] "RemoveContainer" containerID="0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf"
Oct 30 22:32:04.818883 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:04.818863    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf\": container with ID starting with 0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf not found: ID does not exist" containerID="0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf"
Oct 30 22:32:04.818940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.818892    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf"} err="failed to get container status \"0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf\": rpc error: code = NotFound desc = could not find container \"0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf\": container with ID starting with 0a3252b46980f3b795277f6e4766962e696fb9eba71464db68fb767194804daf not found: ID does not exist"
Oct 30 22:32:04.893813 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.893787    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-token\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-token\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.893949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.893836    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4440c1-907a-4752-9a69-197dfcb5ead4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4440c1-907a-4752-9a69-197dfcb5ead4-istiod-ca-cert\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.893949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.893903    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-credential-socket\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.893949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.893942    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-socket\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.894094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.893995    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-data\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.894094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894039    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-kube-api-access-ldd79\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-kube-api-access-ldd79\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.894094 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894072    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-envoy\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.894309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894231    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-credential-socket" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:04.894370 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894281    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-socket" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:04.894436 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894355    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-certs\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.894511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894355    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4440c1-907a-4752-9a69-197dfcb5ead4-istiod-ca-cert" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:04.894511 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894363    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-data" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:04.894621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894537    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-certs" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:04.894621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.894543    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-podinfo\") pod \"de4440c1-907a-4752-9a69-197dfcb5ead4\" (UID: \"de4440c1-907a-4752-9a69-197dfcb5ead4\") "
Oct 30 22:32:04.895416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.895387    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.895553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.895419    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4440c1-907a-4752-9a69-197dfcb5ead4-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.895553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.895435    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.895553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.895450    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.895553 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.895505    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.896459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.896428    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-token" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:04.896593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.896492    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-envoy" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:04.896593 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.896537    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-kube-api-access-ldd79" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "kube-api-access-ldd79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:04.896798 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.896780    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-podinfo" pod "de4440c1-907a-4752-9a69-197dfcb5ead4" (UID: "de4440c1-907a-4752-9a69-197dfcb5ead4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:04.997058 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.996985    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.997058 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.997014    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.997058 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.997023    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldd79\" (UniqueName: \"kubernetes.io/projected/de4440c1-907a-4752-9a69-197dfcb5ead4-kube-api-access-ldd79\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:04.997058 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:04.997034    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4440c1-907a-4752-9a69-197dfcb5ead4-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:05.130800 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:05.130773    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"]
Oct 30 22:32:05.136639 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:05.136613    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c92aec-openshift-default-796f594595-p8h6v"]
Oct 30 22:32:05.465535 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:05.465506    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" path="/var/lib/kubelet/pods/de4440c1-907a-4752-9a69-197dfcb5ead4/volumes"
Oct 30 22:32:06.574377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:06.574343    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"]
Oct 30 22:32:06.589532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:06.589499    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"]
Oct 30 22:32:06.806767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:06.806730    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs" podUID="1be12dee-de32-4f71-baad-46d448718612" containerName="payload-processing" containerID="cri-o://680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe" gracePeriod=25
Oct 30 22:32:06.813338 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:06.813302    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs" podUID="a1a3ae17-f62e-4b49-83a6-6216fd549636" containerName="payload-pre-processing" containerID="cri-o://164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e" gracePeriod=25
Oct 30 22:32:07.525212 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.525188    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:32:07.608222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.608188    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d7feb5-8074-4993-ae62-5352da049922-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d7feb5-8074-4993-ae62-5352da049922-plugins-config-volume\") pod \"31d7feb5-8074-4993-ae62-5352da049922\" (UID: \"31d7feb5-8074-4993-ae62-5352da049922\") "
Oct 30 22:32:07.608702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.608345    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d7feb5-8074-4993-ae62-5352da049922-kube-api-access-hhlfv\" (UniqueName: \"kubernetes.io/projected/31d7feb5-8074-4993-ae62-5352da049922-kube-api-access-hhlfv\") pod \"31d7feb5-8074-4993-ae62-5352da049922\" (UID: \"31d7feb5-8074-4993-ae62-5352da049922\") "
Oct 30 22:32:07.608702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.608618    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d7feb5-8074-4993-ae62-5352da049922-plugins-config-volume" pod "31d7feb5-8074-4993-ae62-5352da049922" (UID: "31d7feb5-8074-4993-ae62-5352da049922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:07.608947 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.608928    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d7feb5-8074-4993-ae62-5352da049922-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:07.610776 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.610751    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d7feb5-8074-4993-ae62-5352da049922-kube-api-access-hhlfv" pod "31d7feb5-8074-4993-ae62-5352da049922" (UID: "31d7feb5-8074-4993-ae62-5352da049922"). InnerVolumeSpecName "kube-api-access-hhlfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:07.710245 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.710219    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlfv\" (UniqueName: \"kubernetes.io/projected/31d7feb5-8074-4993-ae62-5352da049922-kube-api-access-hhlfv\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:07.992999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.992911    4282 generic.go:356] "Generic (PLEG): container finished" podID="31d7feb5-8074-4993-ae62-5352da049922" containerID="015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1" exitCode=0
Oct 30 22:32:07.992999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.992953    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"
Oct 30 22:32:07.993182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.992998    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh" event={"ID":"31d7feb5-8074-4993-ae62-5352da049922","Type":"ContainerDied","Data":"015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1"}
Oct 30 22:32:07.993182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.993058    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh" event={"ID":"31d7feb5-8074-4993-ae62-5352da049922","Type":"ContainerDied","Data":"0199064e6dac92c900ed40b426a00dda8fb8e715b8de7d02a8b79f83c6730963"}
Oct 30 22:32:07.993182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:07.993081    4282 scope.go:122] "RemoveContainer" containerID="015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1"
Oct 30 22:32:08.004498 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:08.004455    4282 scope.go:122] "RemoveContainer" containerID="015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1"
Oct 30 22:32:08.004829 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:08.004809    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1\": container with ID starting with 015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1 not found: ID does not exist" containerID="015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1"
Oct 30 22:32:08.004887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:08.004837    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1"} err="failed to get container status \"015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1\": rpc error: code = NotFound desc = could not find container \"015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1\": container with ID starting with 015e2c06e890c28456987e248bba36e9b0930a82e2e127fd75a35b6d718e6df1 not found: ID does not exist"
Oct 30 22:32:08.017166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:08.017141    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"]
Oct 30 22:32:08.021537 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:08.021514    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-58d5fa-6d46d488c9-lq9rh"]
Oct 30 22:32:09.003983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.003951    4282 generic.go:356] "Generic (PLEG): container finished" podID="303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" containerID="a0356b309a9659226cacda5d202852280ce27b66e719d5cc060f657e3253cd8c" exitCode=0
Oct 30 22:32:09.004389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.004029    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx" event={"ID":"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa","Type":"ContainerDied","Data":"a0356b309a9659226cacda5d202852280ce27b66e719d5cc060f657e3253cd8c"}
Oct 30 22:32:09.007548 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.007526    4282 generic.go:356] "Generic (PLEG): container finished" podID="fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" containerID="48933d7f3cad28e1acf8fea2aeeaa0314669484c87bb6ef224162a510135fc0e" exitCode=0
Oct 30 22:32:09.007681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.007604    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592" event={"ID":"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e","Type":"ContainerDied","Data":"48933d7f3cad28e1acf8fea2aeeaa0314669484c87bb6ef224162a510135fc0e"}
Oct 30 22:32:09.115532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.115503    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:32:09.132938 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.132916    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-plugins-config-volume\") pod \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\" (UID: \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\") "
Oct 30 22:32:09.134575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.132959    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-kube-api-access-nggdf\" (UniqueName: \"kubernetes.io/projected/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-kube-api-access-nggdf\") pod \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\" (UID: \"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e\") "
Oct 30 22:32:09.134575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.133334    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-plugins-config-volume" pod "fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" (UID: "fcb5e5a8-022a-4076-a0c4-4afc2f336c1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:09.134575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.134061    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:09.135971 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.135944    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-kube-api-access-nggdf" pod "fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" (UID: "fcb5e5a8-022a-4076-a0c4-4afc2f336c1e"). InnerVolumeSpecName "kube-api-access-nggdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:09.154731 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.154712    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:32:09.234526 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.234497    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-kube-api-access-x695n\" (UniqueName: \"kubernetes.io/projected/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-kube-api-access-x695n\") pod \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\" (UID: \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\") "
Oct 30 22:32:09.234717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.234601    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-plugins-config-volume\") pod \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\" (UID: \"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa\") "
Oct 30 22:32:09.235061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.235029    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-plugins-config-volume" pod "303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" (UID: "303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:09.235182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.235117    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nggdf\" (UniqueName: \"kubernetes.io/projected/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e-kube-api-access-nggdf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:09.235182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.235133    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:09.236880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.236861    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-kube-api-access-x695n" pod "303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" (UID: "303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa"). InnerVolumeSpecName "kube-api-access-x695n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:09.336241 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.336169    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x695n\" (UniqueName: \"kubernetes.io/projected/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa-kube-api-access-x695n\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:09.466010 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:09.465976    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d7feb5-8074-4993-ae62-5352da049922" path="/var/lib/kubelet/pods/31d7feb5-8074-4993-ae62-5352da049922/volumes"
Oct 30 22:32:10.016812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.016786    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"
Oct 30 22:32:10.016812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.016808    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx" event={"ID":"303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa","Type":"ContainerDied","Data":"1bef3bfc79054080e46e9706912ac275af61eba0ab1364bafc5afdf661c01404"}
Oct 30 22:32:10.017684 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.016864    4282 scope.go:122] "RemoveContainer" containerID="a0356b309a9659226cacda5d202852280ce27b66e719d5cc060f657e3253cd8c"
Oct 30 22:32:10.018355 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.018337    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"
Oct 30 22:32:10.018510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.018363    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592" event={"ID":"fcb5e5a8-022a-4076-a0c4-4afc2f336c1e","Type":"ContainerDied","Data":"21c9703389b0fc5b7764d19ae8d2cf0c716a0645bbea4ab9bcba0a80dc8b85ab"}
Oct 30 22:32:10.027989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.027970    4282 scope.go:122] "RemoveContainer" containerID="48933d7f3cad28e1acf8fea2aeeaa0314669484c87bb6ef224162a510135fc0e"
Oct 30 22:32:10.037124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.037093    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"]
Oct 30 22:32:10.041294 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.041274    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c92aec-7bfdb646df-kc592"]
Oct 30 22:32:10.050713 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.050658    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"]
Oct 30 22:32:10.054067 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.054049    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c92aec-979759b67-74frx"]
Oct 30 22:32:10.063171 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.063150    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"]
Oct 30 22:32:10.077612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:10.077591    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"]
Oct 30 22:32:11.465175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:11.465147    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" path="/var/lib/kubelet/pods/303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa/volumes"
Oct 30 22:32:11.465727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:11.465705    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" path="/var/lib/kubelet/pods/fcb5e5a8-022a-4076-a0c4-4afc2f336c1e/volumes"
Oct 30 22:32:11.575156 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:11.575124    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln" podUID="bf37c164-c4c7-4e73-b615-bb1b9ae62dea" containerName="payload-processing" containerID="cri-o://46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935" gracePeriod=25
Oct 30 22:32:11.589829 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:11.589779    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x" podUID="f7568c7b-2b8f-41d2-8554-e33f16cbf42f" containerName="payload-pre-processing" containerID="cri-o://6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808" gracePeriod=25
Oct 30 22:32:11.598522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:11.598490    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:32:11.687757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:11.687730    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs" podUID="a1a3ae17-f62e-4b49-83a6-6216fd549636" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:12.697550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:12.697513    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs" podUID="1be12dee-de32-4f71-baad-46d448718612" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:13.929280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:13.929240    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x" podUID="f7568c7b-2b8f-41d2-8554-e33f16cbf42f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:13.929280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:13.929268    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln" podUID="bf37c164-c4c7-4e73-b615-bb1b9ae62dea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:15.091653 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:15.091606    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55" podUID="cc88f186-1a26-4dde-a4b3-2cda4e3f218e" containerName="payload-processing" containerID="cri-o://27fbf220cfdee75eb2802990bbb01615208f8c53d7eb13a55ff9ecc721edf5ef" gracePeriod=25
Oct 30 22:32:15.100505 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:15.092525    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29" podUID="07b0d53f-2a75-460e-89f9-a8c2a03608de" containerName="payload-pre-processing" containerID="cri-o://73b4828efe51643513de9e09b8a4b24523559fe30af62b773740c7258133ffda" gracePeriod=25
Oct 30 22:32:16.612723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:16.612689    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55" podUID="cc88f186-1a26-4dde-a4b3-2cda4e3f218e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:16.613161 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:16.612753    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29" podUID="07b0d53f-2a75-460e-89f9-a8c2a03608de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:21.725615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:21.725571    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs" podUID="a1a3ae17-f62e-4b49-83a6-6216fd549636" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:22.697295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:22.697258    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs" podUID="1be12dee-de32-4f71-baad-46d448718612" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:23.929256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:23.929220    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x" podUID="f7568c7b-2b8f-41d2-8554-e33f16cbf42f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:23.929256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:23.929249    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln" podUID="bf37c164-c4c7-4e73-b615-bb1b9ae62dea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:24.109581 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:24.109542    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"]
Oct 30 22:32:24.109841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:24.109811    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy" containerID="cri-o://e4f1bec4abda328466055660d7a30a9e54a6efe88ef09edc271a037ae923a851" gracePeriod=30
Oct 30 22:32:26.612525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:26.612490    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55" podUID="cc88f186-1a26-4dde-a4b3-2cda4e3f218e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:26.612940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:26.612497    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29" podUID="07b0d53f-2a75-460e-89f9-a8c2a03608de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:27.105191 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.105163    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:32:27.148060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.148000    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:32:27.213119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.213074    4282 generic.go:356] "Generic (PLEG): container finished" podID="1be12dee-de32-4f71-baad-46d448718612" containerID="680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe" exitCode=0
Oct 30 22:32:27.213291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.213137    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"
Oct 30 22:32:27.213291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.213208    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs" event={"ID":"1be12dee-de32-4f71-baad-46d448718612","Type":"ContainerDied","Data":"680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe"}
Oct 30 22:32:27.213291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.213257    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs" event={"ID":"1be12dee-de32-4f71-baad-46d448718612","Type":"ContainerDied","Data":"07270b374f45d03900f137bdbe1aab0711149074d4150a9fb88221a17b749f20"}
Oct 30 22:32:27.213291 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.213287    4282 scope.go:122] "RemoveContainer" containerID="680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe"
Oct 30 22:32:27.215215 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.215183    4282 generic.go:356] "Generic (PLEG): container finished" podID="a1a3ae17-f62e-4b49-83a6-6216fd549636" containerID="164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e" exitCode=0
Oct 30 22:32:27.215360 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.215286    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"
Oct 30 22:32:27.215360 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.215331    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs" event={"ID":"a1a3ae17-f62e-4b49-83a6-6216fd549636","Type":"ContainerDied","Data":"164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e"}
Oct 30 22:32:27.215516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.215375    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs" event={"ID":"a1a3ae17-f62e-4b49-83a6-6216fd549636","Type":"ContainerDied","Data":"c5cf76ced6d302f48448560cf17ab911cce823ee4c73dca5ad660e21aa6caa41"}
Oct 30 22:32:27.217482 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.217442    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1be12dee-de32-4f71-baad-46d448718612-kube-api-access-ts6r5\" (UniqueName: \"kubernetes.io/projected/1be12dee-de32-4f71-baad-46d448718612-kube-api-access-ts6r5\") pod \"1be12dee-de32-4f71-baad-46d448718612\" (UID: \"1be12dee-de32-4f71-baad-46d448718612\") "
Oct 30 22:32:27.217617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.217525    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1be12dee-de32-4f71-baad-46d448718612-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be12dee-de32-4f71-baad-46d448718612-plugins-config-volume\") pod \"1be12dee-de32-4f71-baad-46d448718612\" (UID: \"1be12dee-de32-4f71-baad-46d448718612\") "
Oct 30 22:32:27.217692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.217640    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a3ae17-f62e-4b49-83a6-6216fd549636-kube-api-access-5bd8d\" (UniqueName: \"kubernetes.io/projected/a1a3ae17-f62e-4b49-83a6-6216fd549636-kube-api-access-5bd8d\") pod \"a1a3ae17-f62e-4b49-83a6-6216fd549636\" (UID: \"a1a3ae17-f62e-4b49-83a6-6216fd549636\") "
Oct 30 22:32:27.217760 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.217690    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1a3ae17-f62e-4b49-83a6-6216fd549636-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a3ae17-f62e-4b49-83a6-6216fd549636-plugins-config-volume\") pod \"a1a3ae17-f62e-4b49-83a6-6216fd549636\" (UID: \"a1a3ae17-f62e-4b49-83a6-6216fd549636\") "
Oct 30 22:32:27.218200 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.218174    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be12dee-de32-4f71-baad-46d448718612-plugins-config-volume" pod "1be12dee-de32-4f71-baad-46d448718612" (UID: "1be12dee-de32-4f71-baad-46d448718612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:27.218761 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.218737    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1a3ae17-f62e-4b49-83a6-6216fd549636-plugins-config-volume" pod "a1a3ae17-f62e-4b49-83a6-6216fd549636" (UID: "a1a3ae17-f62e-4b49-83a6-6216fd549636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:27.218899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.218773    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be12dee-de32-4f71-baad-46d448718612-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:27.220432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.220388    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be12dee-de32-4f71-baad-46d448718612-kube-api-access-ts6r5" pod "1be12dee-de32-4f71-baad-46d448718612" (UID: "1be12dee-de32-4f71-baad-46d448718612"). InnerVolumeSpecName "kube-api-access-ts6r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:27.221114 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.221077    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a3ae17-f62e-4b49-83a6-6216fd549636-kube-api-access-5bd8d" pod "a1a3ae17-f62e-4b49-83a6-6216fd549636" (UID: "a1a3ae17-f62e-4b49-83a6-6216fd549636"). InnerVolumeSpecName "kube-api-access-5bd8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:27.226400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.226383    4282 scope.go:122] "RemoveContainer" containerID="680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe"
Oct 30 22:32:27.226681 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:27.226665    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe\": container with ID starting with 680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe not found: ID does not exist" containerID="680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe"
Oct 30 22:32:27.226773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.226687    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe"} err="failed to get container status \"680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe\": rpc error: code = NotFound desc = could not find container \"680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe\": container with ID starting with 680f268c6410813adb9b69e1693dad324a9a4e9c6c100a09b03ecb84b80903fe not found: ID does not exist"
Oct 30 22:32:27.226773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.226702    4282 scope.go:122] "RemoveContainer" containerID="164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e"
Oct 30 22:32:27.236274 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.236257    4282 scope.go:122] "RemoveContainer" containerID="164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e"
Oct 30 22:32:27.236590 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:27.236572    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e\": container with ID starting with 164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e not found: ID does not exist" containerID="164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e"
Oct 30 22:32:27.236664 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.236600    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e"} err="failed to get container status \"164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e\": rpc error: code = NotFound desc = could not find container \"164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e\": container with ID starting with 164f7fc9752159691527979a2a0be0895451d9ef067dd6d8b3629d204126443e not found: ID does not exist"
Oct 30 22:32:27.324489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.324192    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1a3ae17-f62e-4b49-83a6-6216fd549636-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:27.324489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.324233    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6r5\" (UniqueName: \"kubernetes.io/projected/1be12dee-de32-4f71-baad-46d448718612-kube-api-access-ts6r5\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:27.324489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.324247    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bd8d\" (UniqueName: \"kubernetes.io/projected/a1a3ae17-f62e-4b49-83a6-6216fd549636-kube-api-access-5bd8d\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:27.531454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.531419    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"]
Oct 30 22:32:27.535261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.535235    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4b2bb0-b4b86c699-klfjs"]
Oct 30 22:32:27.546643 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.546618    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"]
Oct 30 22:32:27.550453 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:27.550431    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4b2bb0-796f9c94c-zs6qs"]
Oct 30 22:32:29.234070 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.234044    4282 generic.go:356] "Generic (PLEG): container finished" podID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerID="e4f1bec4abda328466055660d7a30a9e54a6efe88ef09edc271a037ae923a851" exitCode=0
Oct 30 22:32:29.234510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.234111    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" event={"ID":"3e713fa3-0235-4cb6-aea0-65fa0591293d","Type":"ContainerDied","Data":"e4f1bec4abda328466055660d7a30a9e54a6efe88ef09edc271a037ae923a851"}
Oct 30 22:32:29.234510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.234165    4282 scope.go:122] "RemoveContainer" containerID="df7e3128ba0207a81ce8384143609033ee1230c7855c8d0b1a0cab08bd18d311"
Oct 30 22:32:29.403919 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.403897    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:32:29.484375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.484333    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1be12dee-de32-4f71-baad-46d448718612" path="/var/lib/kubelet/pods/1be12dee-de32-4f71-baad-46d448718612/volumes"
Oct 30 22:32:29.485248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.485218    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1a3ae17-f62e-4b49-83a6-6216fd549636" path="/var/lib/kubelet/pods/a1a3ae17-f62e-4b49-83a6-6216fd549636/volumes"
Oct 30 22:32:29.587415 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587379    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e713fa3-0235-4cb6-aea0-65fa0591293d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e713fa3-0235-4cb6-aea0-65fa0591293d-istiod-ca-cert\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.587655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587434    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-certs\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.587655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587544    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-data\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.587655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587619    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-socket\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.587842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587675    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-credential-socket\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.587842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587723    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-token\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-token\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.587842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587750    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-podinfo\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.587842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587769    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e713fa3-0235-4cb6-aea0-65fa0591293d-istiod-ca-cert" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:29.587842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587799    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-certs" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:29.587842 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587836    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-kube-api-access-8wpsq\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-kube-api-access-8wpsq\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.588180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587889    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-envoy\") pod \"3e713fa3-0235-4cb6-aea0-65fa0591293d\" (UID: \"3e713fa3-0235-4cb6-aea0-65fa0591293d\") "
Oct 30 22:32:29.588180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587866    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-data" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:29.588180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587990    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-credential-socket" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:29.588180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.587942    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-socket" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:29.588804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.588782    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:29.588899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.588809    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:29.588899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.588825    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:29.588899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.588840    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e713fa3-0235-4cb6-aea0-65fa0591293d-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:29.588899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.588853    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:29.590228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.590205    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-token" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:29.590322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.590227    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-kube-api-access-8wpsq" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "kube-api-access-8wpsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:29.590642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.590624    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-podinfo" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:29.590642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.590628    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-envoy" pod "3e713fa3-0235-4cb6-aea0-65fa0591293d" (UID: "3e713fa3-0235-4cb6-aea0-65fa0591293d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:29.690039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.690003    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:29.690039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.690034    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:29.690039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.690044    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wpsq\" (UniqueName: \"kubernetes.io/projected/3e713fa3-0235-4cb6-aea0-65fa0591293d-kube-api-access-8wpsq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:29.690287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:29.690054    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e713fa3-0235-4cb6-aea0-65fa0591293d-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:30.242989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:30.242960    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"
Oct 30 22:32:30.242989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:30.242972    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww" event={"ID":"3e713fa3-0235-4cb6-aea0-65fa0591293d","Type":"ContainerDied","Data":"c3b72cde4f8334bb3574831a1789084bdeb18f21477e968476dbff40d1da6fd0"}
Oct 30 22:32:30.243547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:30.243033    4282 scope.go:122] "RemoveContainer" containerID="e4f1bec4abda328466055660d7a30a9e54a6efe88ef09edc271a037ae923a851"
Oct 30 22:32:30.266840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:30.266812    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"]
Oct 30 22:32:30.272660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:30.272634    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bacea0-openshift-default-6f6786548f-bq8ww"]
Oct 30 22:32:31.384462 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:31.384418    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"]
Oct 30 22:32:31.467983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:31.467948    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" path="/var/lib/kubelet/pods/3e713fa3-0235-4cb6-aea0-65fa0591293d/volumes"
Oct 30 22:32:31.896804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:31.896738    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"]
Oct 30 22:32:31.897018 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:31.896977    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr" podUID="84b39bf1-07b7-46c4-bcf7-3b955a955bd8" containerName="maas-api" containerID="cri-o://f4bedda2f34f0923475422c16af6fd1f61aa1165651b87050695a12889d67eeb" gracePeriod=30
Oct 30 22:32:32.033786 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.033756    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:32:32.038825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.038802    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:32:32.157691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.153343    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-plugins-config-volume\") pod \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\" (UID: \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\") "
Oct 30 22:32:32.157691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.153419    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-plugins-config-volume\") pod \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\" (UID: \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\") "
Oct 30 22:32:32.157691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.153508    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-kube-api-access-rckjj\" (UniqueName: \"kubernetes.io/projected/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-kube-api-access-rckjj\") pod \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\" (UID: \"f7568c7b-2b8f-41d2-8554-e33f16cbf42f\") "
Oct 30 22:32:32.157691 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.153627    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-kube-api-access-s8vcf\" (UniqueName: \"kubernetes.io/projected/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-kube-api-access-s8vcf\") pod \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\" (UID: \"bf37c164-c4c7-4e73-b615-bb1b9ae62dea\") "
Oct 30 22:32:32.201652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.191678    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-plugins-config-volume" pod "bf37c164-c4c7-4e73-b615-bb1b9ae62dea" (UID: "bf37c164-c4c7-4e73-b615-bb1b9ae62dea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:32.212922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.211391    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-plugins-config-volume" pod "f7568c7b-2b8f-41d2-8554-e33f16cbf42f" (UID: "f7568c7b-2b8f-41d2-8554-e33f16cbf42f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:32.240850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.237387    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-kube-api-access-s8vcf" pod "bf37c164-c4c7-4e73-b615-bb1b9ae62dea" (UID: "bf37c164-c4c7-4e73-b615-bb1b9ae62dea"). InnerVolumeSpecName "kube-api-access-s8vcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:32.240850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.240282    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-kube-api-access-rckjj" pod "f7568c7b-2b8f-41d2-8554-e33f16cbf42f" (UID: "f7568c7b-2b8f-41d2-8554-e33f16cbf42f"). InnerVolumeSpecName "kube-api-access-rckjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:32.294996 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.294955    4282 generic.go:356] "Generic (PLEG): container finished" podID="f7568c7b-2b8f-41d2-8554-e33f16cbf42f" containerID="6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808" exitCode=0
Oct 30 22:32:32.295175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.295065    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x" event={"ID":"f7568c7b-2b8f-41d2-8554-e33f16cbf42f","Type":"ContainerDied","Data":"6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808"}
Oct 30 22:32:32.295175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.295116    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x" event={"ID":"f7568c7b-2b8f-41d2-8554-e33f16cbf42f","Type":"ContainerDied","Data":"aa44b0215827c3672695ed261b478c37de559f332811fc31653e5fc7b71f7fea"}
Oct 30 22:32:32.295175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.295145    4282 scope.go:122] "RemoveContainer" containerID="6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808"
Oct 30 22:32:32.295377 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.295261    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"
Oct 30 22:32:32.300529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.300152    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:32.300529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.300224    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:32.300529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.300242    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rckjj\" (UniqueName: \"kubernetes.io/projected/f7568c7b-2b8f-41d2-8554-e33f16cbf42f-kube-api-access-rckjj\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:32.300529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.300259    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8vcf\" (UniqueName: \"kubernetes.io/projected/bf37c164-c4c7-4e73-b615-bb1b9ae62dea-kube-api-access-s8vcf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:32.302354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.302137    4282 generic.go:356] "Generic (PLEG): container finished" podID="84b39bf1-07b7-46c4-bcf7-3b955a955bd8" containerID="f4bedda2f34f0923475422c16af6fd1f61aa1165651b87050695a12889d67eeb" exitCode=0
Oct 30 22:32:32.302354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.302294    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr" event={"ID":"84b39bf1-07b7-46c4-bcf7-3b955a955bd8","Type":"ContainerDied","Data":"f4bedda2f34f0923475422c16af6fd1f61aa1165651b87050695a12889d67eeb"}
Oct 30 22:32:32.306260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.305058    4282 generic.go:356] "Generic (PLEG): container finished" podID="bf37c164-c4c7-4e73-b615-bb1b9ae62dea" containerID="46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935" exitCode=0
Oct 30 22:32:32.306260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.305194    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln" event={"ID":"bf37c164-c4c7-4e73-b615-bb1b9ae62dea","Type":"ContainerDied","Data":"46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935"}
Oct 30 22:32:32.306260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.305236    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln" event={"ID":"bf37c164-c4c7-4e73-b615-bb1b9ae62dea","Type":"ContainerDied","Data":"d0c53149e8872c17e189c569639f1f524581126b10d667be1c1a51e6855ec7ed"}
Oct 30 22:32:32.306260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.305305    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"
Oct 30 22:32:32.312804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.312785    4282 scope.go:122] "RemoveContainer" containerID="6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808"
Oct 30 22:32:32.313158 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:32.313131    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808\": container with ID starting with 6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808 not found: ID does not exist" containerID="6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808"
Oct 30 22:32:32.313228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.313169    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808"} err="failed to get container status \"6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808\": rpc error: code = NotFound desc = could not find container \"6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808\": container with ID starting with 6f326621af815dbc70a5a57c1b2250f3650b4c02178ac9c4f82c295d0d2e0808 not found: ID does not exist"
Oct 30 22:32:32.313228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.313192    4282 scope.go:122] "RemoveContainer" containerID="46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935"
Oct 30 22:32:32.325116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.325094    4282 scope.go:122] "RemoveContainer" containerID="46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935"
Oct 30 22:32:32.325318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.325296    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"]
Oct 30 22:32:32.325419 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:32.325387    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935\": container with ID starting with 46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935 not found: ID does not exist" containerID="46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935"
Oct 30 22:32:32.325521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.325414    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935"} err="failed to get container status \"46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935\": rpc error: code = NotFound desc = could not find container \"46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935\": container with ID starting with 46bf1f97fc6bf70fc29f5b222a43de1760926f849273cf945570ca376bb59935 not found: ID does not exist"
Oct 30 22:32:32.329427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.329405    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bacea0-64bcfddf59-k2k5x"]
Oct 30 22:32:32.335641 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.335623    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:32:32.339112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.339092    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"]
Oct 30 22:32:32.344354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.344335    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-bacea0-b7486cb67-cbnln"]
Oct 30 22:32:32.401582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.401534    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-kube-api-access-469jd\" (UniqueName: \"kubernetes.io/projected/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-kube-api-access-469jd\") pod \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") "
Oct 30 22:32:32.402100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.401605    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-metrics-certs\") pod \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") "
Oct 30 22:32:32.402100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.401757    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-openshift-service-ca\") pod \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") "
Oct 30 22:32:32.402100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.401822    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls\") pod \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\" (UID: \"84b39bf1-07b7-46c4-bcf7-3b955a955bd8\") "
Oct 30 22:32:32.402356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.402330    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-openshift-service-ca" pod "84b39bf1-07b7-46c4-bcf7-3b955a955bd8" (UID: "84b39bf1-07b7-46c4-bcf7-3b955a955bd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:32.402583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.402535    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:32.404720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.404682    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-metrics-certs" pod "84b39bf1-07b7-46c4-bcf7-3b955a955bd8" (UID: "84b39bf1-07b7-46c4-bcf7-3b955a955bd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:32.404720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.404692    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls" pod "84b39bf1-07b7-46c4-bcf7-3b955a955bd8" (UID: "84b39bf1-07b7-46c4-bcf7-3b955a955bd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:32.404902 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.404803    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-kube-api-access-469jd" pod "84b39bf1-07b7-46c4-bcf7-3b955a955bd8" (UID: "84b39bf1-07b7-46c4-bcf7-3b955a955bd8"). InnerVolumeSpecName "kube-api-access-469jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:32.511547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.510942    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:32.511547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.510991    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:32.511547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:32.511012    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-469jd\" (UniqueName: \"kubernetes.io/projected/84b39bf1-07b7-46c4-bcf7-3b955a955bd8-kube-api-access-469jd\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:33.316878 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:33.316851    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"
Oct 30 22:32:33.317201 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:33.316847    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr" event={"ID":"84b39bf1-07b7-46c4-bcf7-3b955a955bd8","Type":"ContainerDied","Data":"445e373f6e6fa6e1b02e1635addc0e6403f25641acff3446caa3e26a4093d8b9"}
Oct 30 22:32:33.317201 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:33.316994    4282 scope.go:122] "RemoveContainer" containerID="f4bedda2f34f0923475422c16af6fd1f61aa1165651b87050695a12889d67eeb"
Oct 30 22:32:33.344136 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:33.344108    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"]
Oct 30 22:32:33.348904 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:33.348879    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b9d868-d8dddc576-lvxrr"]
Oct 30 22:32:33.464547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:33.464519    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b39bf1-07b7-46c4-bcf7-3b955a955bd8" path="/var/lib/kubelet/pods/84b39bf1-07b7-46c4-bcf7-3b955a955bd8/volumes"
Oct 30 22:32:33.464965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:33.464951    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf37c164-c4c7-4e73-b615-bb1b9ae62dea" path="/var/lib/kubelet/pods/bf37c164-c4c7-4e73-b615-bb1b9ae62dea/volumes"
Oct 30 22:32:33.465320 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:33.465308    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7568c7b-2b8f-41d2-8554-e33f16cbf42f" path="/var/lib/kubelet/pods/f7568c7b-2b8f-41d2-8554-e33f16cbf42f/volumes"
Oct 30 22:32:34.678316 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:34.678274    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:32:34.749549 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:34.749507    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"]
Oct 30 22:32:34.775558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:34.775524    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"]
Oct 30 22:32:35.155500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.130910    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"]
Oct 30 22:32:35.155500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.131606    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz" podUID="5b7ec276-992d-40e4-a83e-65b55a7f7df7" containerName="maas-api" containerID="cri-o://49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d" gracePeriod=30
Oct 30 22:32:35.361102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.352500    4282 generic.go:356] "Generic (PLEG): container finished" podID="cc88f186-1a26-4dde-a4b3-2cda4e3f218e" containerID="27fbf220cfdee75eb2802990bbb01615208f8c53d7eb13a55ff9ecc721edf5ef" exitCode=0
Oct 30 22:32:35.361102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.352677    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55" event={"ID":"cc88f186-1a26-4dde-a4b3-2cda4e3f218e","Type":"ContainerDied","Data":"27fbf220cfdee75eb2802990bbb01615208f8c53d7eb13a55ff9ecc721edf5ef"}
Oct 30 22:32:35.361102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.354922    4282 generic.go:356] "Generic (PLEG): container finished" podID="07b0d53f-2a75-460e-89f9-a8c2a03608de" containerID="73b4828efe51643513de9e09b8a4b24523559fe30af62b773740c7258133ffda" exitCode=0
Oct 30 22:32:35.361102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.355017    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29" event={"ID":"07b0d53f-2a75-460e-89f9-a8c2a03608de","Type":"ContainerDied","Data":"73b4828efe51643513de9e09b8a4b24523559fe30af62b773740c7258133ffda"}
Oct 30 22:32:35.685393 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.684836    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:32:35.687483 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.687429    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:32:35.779101 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.779066    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:32:35.814963 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.814934    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-kube-api-access-ljcnv\" (UniqueName: \"kubernetes.io/projected/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-kube-api-access-ljcnv\") pod \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\" (UID: \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\") "
Oct 30 22:32:35.815136 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.815125    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b0d53f-2a75-460e-89f9-a8c2a03608de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0d53f-2a75-460e-89f9-a8c2a03608de-plugins-config-volume\") pod \"07b0d53f-2a75-460e-89f9-a8c2a03608de\" (UID: \"07b0d53f-2a75-460e-89f9-a8c2a03608de\") "
Oct 30 22:32:35.815198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.815188    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-plugins-config-volume\") pod \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\" (UID: \"cc88f186-1a26-4dde-a4b3-2cda4e3f218e\") "
Oct 30 22:32:35.815247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.815214    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b0d53f-2a75-460e-89f9-a8c2a03608de-kube-api-access-twr4k\" (UniqueName: \"kubernetes.io/projected/07b0d53f-2a75-460e-89f9-a8c2a03608de-kube-api-access-twr4k\") pod \"07b0d53f-2a75-460e-89f9-a8c2a03608de\" (UID: \"07b0d53f-2a75-460e-89f9-a8c2a03608de\") "
Oct 30 22:32:35.816413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.816383    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-plugins-config-volume" pod "cc88f186-1a26-4dde-a4b3-2cda4e3f218e" (UID: "cc88f186-1a26-4dde-a4b3-2cda4e3f218e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:35.816601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.816568    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b0d53f-2a75-460e-89f9-a8c2a03608de-plugins-config-volume" pod "07b0d53f-2a75-460e-89f9-a8c2a03608de" (UID: "07b0d53f-2a75-460e-89f9-a8c2a03608de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:35.819053 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.819025    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b0d53f-2a75-460e-89f9-a8c2a03608de-kube-api-access-twr4k" pod "07b0d53f-2a75-460e-89f9-a8c2a03608de" (UID: "07b0d53f-2a75-460e-89f9-a8c2a03608de"). InnerVolumeSpecName "kube-api-access-twr4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:35.820520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.820315    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-kube-api-access-ljcnv" pod "cc88f186-1a26-4dde-a4b3-2cda4e3f218e" (UID: "cc88f186-1a26-4dde-a4b3-2cda4e3f218e"). InnerVolumeSpecName "kube-api-access-ljcnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:35.919893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.919539    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-maas-api-tls\") pod \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") "
Oct 30 22:32:35.919893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.919679    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b7ec276-992d-40e4-a83e-65b55a7f7df7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7ec276-992d-40e4-a83e-65b55a7f7df7-openshift-service-ca\") pod \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") "
Oct 30 22:32:35.919893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.919805    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b7ec276-992d-40e4-a83e-65b55a7f7df7-kube-api-access-z4x9h\" (UniqueName: \"kubernetes.io/projected/5b7ec276-992d-40e4-a83e-65b55a7f7df7-kube-api-access-z4x9h\") pod \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") "
Oct 30 22:32:35.919893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.919867    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-metrics-certs\") pod \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\" (UID: \"5b7ec276-992d-40e4-a83e-65b55a7f7df7\") "
Oct 30 22:32:35.920620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.920585    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b7ec276-992d-40e4-a83e-65b55a7f7df7-openshift-service-ca" pod "5b7ec276-992d-40e4-a83e-65b55a7f7df7" (UID: "5b7ec276-992d-40e4-a83e-65b55a7f7df7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:35.920876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.920853    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b7ec276-992d-40e4-a83e-65b55a7f7df7-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:35.920965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.920881    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b0d53f-2a75-460e-89f9-a8c2a03608de-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:35.920965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.920897    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:35.920965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.920911    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twr4k\" (UniqueName: \"kubernetes.io/projected/07b0d53f-2a75-460e-89f9-a8c2a03608de-kube-api-access-twr4k\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:35.920965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.920926    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljcnv\" (UniqueName: \"kubernetes.io/projected/cc88f186-1a26-4dde-a4b3-2cda4e3f218e-kube-api-access-ljcnv\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:35.922941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.922912    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b7ec276-992d-40e4-a83e-65b55a7f7df7-kube-api-access-z4x9h" pod "5b7ec276-992d-40e4-a83e-65b55a7f7df7" (UID: "5b7ec276-992d-40e4-a83e-65b55a7f7df7"). InnerVolumeSpecName "kube-api-access-z4x9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:35.923589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.923566    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-maas-api-tls" pod "5b7ec276-992d-40e4-a83e-65b55a7f7df7" (UID: "5b7ec276-992d-40e4-a83e-65b55a7f7df7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:35.924358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:35.924332    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-metrics-certs" pod "5b7ec276-992d-40e4-a83e-65b55a7f7df7" (UID: "5b7ec276-992d-40e4-a83e-65b55a7f7df7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:32:36.021877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.021844    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:36.021877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.021872    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4x9h\" (UniqueName: \"kubernetes.io/projected/5b7ec276-992d-40e4-a83e-65b55a7f7df7-kube-api-access-z4x9h\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:36.021877 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.021882    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b7ec276-992d-40e4-a83e-65b55a7f7df7-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:36.363952 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.363921    4282 generic.go:356] "Generic (PLEG): container finished" podID="5b7ec276-992d-40e4-a83e-65b55a7f7df7" containerID="49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d" exitCode=0
Oct 30 22:32:36.364164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.363981    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"
Oct 30 22:32:36.364164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.364005    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz" event={"ID":"5b7ec276-992d-40e4-a83e-65b55a7f7df7","Type":"ContainerDied","Data":"49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d"}
Oct 30 22:32:36.364164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.364094    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz" event={"ID":"5b7ec276-992d-40e4-a83e-65b55a7f7df7","Type":"ContainerDied","Data":"8d22d195ce8e03c0f4b12daa2d57299cbc104f3dc3268a4796d0c58e52a6b480"}
Oct 30 22:32:36.364164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.364128    4282 scope.go:122] "RemoveContainer" containerID="49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d"
Oct 30 22:32:36.365800 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.365777    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"
Oct 30 22:32:36.366271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.366005    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55" event={"ID":"cc88f186-1a26-4dde-a4b3-2cda4e3f218e","Type":"ContainerDied","Data":"092ce6eb7892c3f6110cf2f295541d622f910774cbd0fe18a2e39488c03176c2"}
Oct 30 22:32:36.367920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.367897    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29" event={"ID":"07b0d53f-2a75-460e-89f9-a8c2a03608de","Type":"ContainerDied","Data":"0e7f5a7eaa8ea6da006843d9b062e75e9dee08efe27a74733bd1a9d327a7dfc1"}
Oct 30 22:32:36.367920 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.367916    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"
Oct 30 22:32:36.378233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.378212    4282 scope.go:122] "RemoveContainer" containerID="49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d"
Oct 30 22:32:36.378581 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:36.378555    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d\": container with ID starting with 49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d not found: ID does not exist" containerID="49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d"
Oct 30 22:32:36.378692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.378593    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d"} err="failed to get container status \"49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d\": rpc error: code = NotFound desc = could not find container \"49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d\": container with ID starting with 49ba11c1b9cab0c6b9390a95f6b7068d88cdbfe389567bc3c4e91cd8dd4b0d9d not found: ID does not exist"
Oct 30 22:32:36.378692 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.378616    4282 scope.go:122] "RemoveContainer" containerID="27fbf220cfdee75eb2802990bbb01615208f8c53d7eb13a55ff9ecc721edf5ef"
Oct 30 22:32:36.384903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.384876    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv" podUID="e94ea251-952e-4a62-bb7d-c39f389542ee" containerName="payload-pre-processing" containerID="cri-o://5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d" gracePeriod=25
Oct 30 22:32:36.390503 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.390449    4282 scope.go:122] "RemoveContainer" containerID="73b4828efe51643513de9e09b8a4b24523559fe30af62b773740c7258133ffda"
Oct 30 22:32:36.392967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.392946    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"]
Oct 30 22:32:36.396837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.396814    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-28b4bb-7964966d44-8njrz"]
Oct 30 22:32:36.429459 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.427801    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"]
Oct 30 22:32:36.448632 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.448551    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5bdf-84f6474f7f-wdh29"]
Oct 30 22:32:36.450223 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.450200    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"]
Oct 30 22:32:36.453308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:36.453289    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5bdf-6d8d4789f7-9wk55"]
Oct 30 22:32:37.464871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:37.464833    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b0d53f-2a75-460e-89f9-a8c2a03608de" path="/var/lib/kubelet/pods/07b0d53f-2a75-460e-89f9-a8c2a03608de/volumes"
Oct 30 22:32:37.465532 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:37.465513    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b7ec276-992d-40e4-a83e-65b55a7f7df7" path="/var/lib/kubelet/pods/5b7ec276-992d-40e4-a83e-65b55a7f7df7/volumes"
Oct 30 22:32:37.465965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:37.465946    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc88f186-1a26-4dde-a4b3-2cda4e3f218e" path="/var/lib/kubelet/pods/cc88f186-1a26-4dde-a4b3-2cda4e3f218e/volumes"
Oct 30 22:32:39.750612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:39.750566    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t" podUID="0438efc5-012a-4fb4-98cf-7b8b9af86145" containerName="payload-processing" containerID="cri-o://0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07" gracePeriod=25
Oct 30 22:32:39.776016 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:39.775984    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55" podUID="2486a5ab-245c-40cb-93d0-c01ab6408012" containerName="payload-pre-processing" containerID="cri-o://756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d" gracePeriod=25
Oct 30 22:32:40.346151 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:40.346114    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"]
Oct 30 22:32:40.346398 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:40.346369    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" containerID="cri-o://90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d" gracePeriod=30
Oct 30 22:32:43.399852 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:43.399820    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"]
Oct 30 22:32:43.400365 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:43.400035    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy" containerID="cri-o://fcd8b766b5ce9cb275773b409311c243f507e96fe564d649043aa8dae887596f" gracePeriod=30
Oct 30 22:32:44.410767 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:44.410735    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv" podUID="e94ea251-952e-4a62-bb7d-c39f389542ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:44.856569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:44.856533    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55" podUID="2486a5ab-245c-40cb-93d0-c01ab6408012" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:45.633441 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.633415    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:32:45.659517 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.659485    4282 generic.go:356] "Generic (PLEG): container finished" podID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerID="90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d" exitCode=0
Oct 30 22:32:45.659660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.659610    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" event={"ID":"bd34ab38-6807-4efc-961f-6b10adb0f268","Type":"ContainerDied","Data":"90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d"}
Oct 30 22:32:45.659660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.659657    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" event={"ID":"bd34ab38-6807-4efc-961f-6b10adb0f268","Type":"ContainerDied","Data":"41ab5f335d95e782ae76a5a98b3e738fa466ee4ce79d4bf6497020c6697ba45f"}
Oct 30 22:32:45.659789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.659685    4282 scope.go:122] "RemoveContainer" containerID="90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d"
Oct 30 22:32:45.659859 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.659841    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"
Oct 30 22:32:45.672817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.672794    4282 scope.go:122] "RemoveContainer" containerID="4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d"
Oct 30 22:32:45.686808 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.686780    4282 scope.go:122] "RemoveContainer" containerID="90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d"
Oct 30 22:32:45.688638 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:45.688609    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d\": container with ID starting with 90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d not found: ID does not exist" containerID="90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d"
Oct 30 22:32:45.688759 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.688649    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d"} err="failed to get container status \"90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d\": rpc error: code = NotFound desc = could not find container \"90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d\": container with ID starting with 90134722e08f5a58acd4478760e71b9f240740f67c14668590a6c54ef002459d not found: ID does not exist"
Oct 30 22:32:45.688759 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.688672    4282 scope.go:122] "RemoveContainer" containerID="4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d"
Oct 30 22:32:45.689328 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:45.689299    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d\": container with ID starting with 4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d not found: ID does not exist" containerID="4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d"
Oct 30 22:32:45.689434 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.689337    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d"} err="failed to get container status \"4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d\": rpc error: code = NotFound desc = could not find container \"4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d\": container with ID starting with 4a3f47341fb449b6e0ce58d9c924e1659e1bf56f51e97695c6b8b7404ebc9f6d not found: ID does not exist"
Oct 30 22:32:45.744038 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744007    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-podinfo\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.744168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744060    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-kube-api-access-6dkrk\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-kube-api-access-6dkrk\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.744168 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744086    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-credential-socket\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.744339 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744315    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-envoy\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.744412 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744394    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-credential-socket" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.744412 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744403    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd34ab38-6807-4efc-961f-6b10adb0f268-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd34ab38-6807-4efc-961f-6b10adb0f268-istiod-ca-cert\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.744572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744523    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-token\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-token\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.744662 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744643    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-socket\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.744760 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744688    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-data\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.744760 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744749    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd34ab38-6807-4efc-961f-6b10adb0f268-istiod-ca-cert" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:45.745122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744778    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-certs\") pod \"bd34ab38-6807-4efc-961f-6b10adb0f268\" (UID: \"bd34ab38-6807-4efc-961f-6b10adb0f268\") "
Oct 30 22:32:45.745122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.744900    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-socket" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.745122 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.745003    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-data" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.745681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.745644    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-certs" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.745973 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.745947    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.746072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.745981    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.746072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.745996    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.746072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.746011    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.746072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.746027    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd34ab38-6807-4efc-961f-6b10adb0f268-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.746618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.746562    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-podinfo" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:45.746855 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.746830    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-kube-api-access-6dkrk" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "kube-api-access-6dkrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:45.747026 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.747007    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-envoy" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:45.747124 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.747109    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-token" pod "bd34ab38-6807-4efc-961f-6b10adb0f268" (UID: "bd34ab38-6807-4efc-961f-6b10adb0f268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:45.854068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.854024    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.854068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.854068    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dkrk\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-kube-api-access-6dkrk\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.854068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.854082    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.854068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.854095    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd34ab38-6807-4efc-961f-6b10adb0f268-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:45.984268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.984238    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"]
Oct 30 22:32:45.996420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:45.996380    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7"]
Oct 30 22:32:46.146899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:46.146868    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t" podUID="0438efc5-012a-4fb4-98cf-7b8b9af86145" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:46.559487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:46.557260    4282 patch_prober.go:29] interesting pod/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:32:46.559487 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:46.557346    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-b9d868-openshift-default-7669748cc-vqqz7" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:32:47.465005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:47.464973    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" path="/var/lib/kubelet/pods/bd34ab38-6807-4efc-961f-6b10adb0f268/volumes"
Oct 30 22:32:48.683589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.683561    4282 generic.go:356] "Generic (PLEG): container finished" podID="716883c7-6993-4ea2-b694-f334aba562c0" containerID="fcd8b766b5ce9cb275773b409311c243f507e96fe564d649043aa8dae887596f" exitCode=0
Oct 30 22:32:48.683989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.683641    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" event={"ID":"716883c7-6993-4ea2-b694-f334aba562c0","Type":"ContainerDied","Data":"fcd8b766b5ce9cb275773b409311c243f507e96fe564d649043aa8dae887596f"}
Oct 30 22:32:48.683989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.683712    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc" event={"ID":"716883c7-6993-4ea2-b694-f334aba562c0","Type":"ContainerDied","Data":"1261c177e63168266da69d486df8d5b624d1247101ecf2bace97921b17a6ac56"}
Oct 30 22:32:48.683989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.683740    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1261c177e63168266da69d486df8d5b624d1247101ecf2bace97921b17a6ac56"
Oct 30 22:32:48.684886 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.684871    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:32:48.699041 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699016    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-data\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699073    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-istio-token\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-istio-token\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699123    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-socket\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699161    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-credential-socket\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699289    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-data" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:48.699317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699291    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-kube-api-access-dlhnp\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-kube-api-access-dlhnp\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699432 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699366    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/716883c7-6993-4ea2-b694-f334aba562c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716883c7-6993-4ea2-b694-f334aba562c0-istio-podinfo\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699447    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-socket" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:48.699580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699512    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-envoy\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699580 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699514    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-credential-socket" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:48.699702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699600    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716883c7-6993-4ea2-b694-f334aba562c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716883c7-6993-4ea2-b694-f334aba562c0-istiod-ca-cert\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699682    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-certs\") pod \"716883c7-6993-4ea2-b694-f334aba562c0\" (UID: \"716883c7-6993-4ea2-b694-f334aba562c0\") "
Oct 30 22:32:48.699987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699938    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-certs" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:48.699987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.699956    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716883c7-6993-4ea2-b694-f334aba562c0-istiod-ca-cert" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:48.700346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.700289    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/716883c7-6993-4ea2-b694-f334aba562c0-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.700346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.700310    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.700346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.700319    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.700346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.700328    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.700346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.700337    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.701817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.701776    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-envoy" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:48.701911 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.701867    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/716883c7-6993-4ea2-b694-f334aba562c0-istio-podinfo" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:32:48.701958 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.701944    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-kube-api-access-dlhnp" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "kube-api-access-dlhnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:48.702154 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.702139    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-istio-token" pod "716883c7-6993-4ea2-b694-f334aba562c0" (UID: "716883c7-6993-4ea2-b694-f334aba562c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:48.801399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.801358    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.801399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.801386    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhnp\" (UniqueName: \"kubernetes.io/projected/716883c7-6993-4ea2-b694-f334aba562c0-kube-api-access-dlhnp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.801399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.801398    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/716883c7-6993-4ea2-b694-f334aba562c0-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.801399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.801407    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/716883c7-6993-4ea2-b694-f334aba562c0-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:48.874369 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:48.874335    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"]
Oct 30 22:32:49.689744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:49.689716    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"
Oct 30 22:32:49.710243 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:49.710214    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"]
Oct 30 22:32:49.715257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:49.715236    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-fplsc"]
Oct 30 22:32:50.984153 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:50.984113    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"]
Oct 30 22:32:50.994015 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:50.993988    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"]
Oct 30 22:32:51.464406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:51.464373    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716883c7-6993-4ea2-b694-f334aba562c0" path="/var/lib/kubelet/pods/716883c7-6993-4ea2-b694-f334aba562c0/volumes"
Oct 30 22:32:54.411241 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:54.411160    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv" podUID="e94ea251-952e-4a62-bb7d-c39f389542ee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:54.857202 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:54.857166    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55" podUID="2486a5ab-245c-40cb-93d0-c01ab6408012" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:55.986097 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:55.986058    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs" podUID="0688b402-ba16-4049-a38e-ece12d389c7d" containerName="payload-processing" containerID="cri-o://86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03" gracePeriod=25
Oct 30 22:32:55.994560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:55.994513    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp" podUID="ae82a63f-9f4c-4a2a-b713-127e52277966" containerName="payload-pre-processing" containerID="cri-o://6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4" gracePeriod=25
Oct 30 22:32:56.145865 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.145825    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t" podUID="0438efc5-012a-4fb4-98cf-7b8b9af86145" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:32:56.658330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.658262    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:32:56.719522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.719462    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94ea251-952e-4a62-bb7d-c39f389542ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94ea251-952e-4a62-bb7d-c39f389542ee-plugins-config-volume\") pod \"e94ea251-952e-4a62-bb7d-c39f389542ee\" (UID: \"e94ea251-952e-4a62-bb7d-c39f389542ee\") "
Oct 30 22:32:56.719694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.719549    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94ea251-952e-4a62-bb7d-c39f389542ee-kube-api-access-7px22\" (UniqueName: \"kubernetes.io/projected/e94ea251-952e-4a62-bb7d-c39f389542ee-kube-api-access-7px22\") pod \"e94ea251-952e-4a62-bb7d-c39f389542ee\" (UID: \"e94ea251-952e-4a62-bb7d-c39f389542ee\") "
Oct 30 22:32:56.719850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.719826    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94ea251-952e-4a62-bb7d-c39f389542ee-plugins-config-volume" pod "e94ea251-952e-4a62-bb7d-c39f389542ee" (UID: "e94ea251-952e-4a62-bb7d-c39f389542ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:32:56.720134 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.720115    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94ea251-952e-4a62-bb7d-c39f389542ee-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:56.722137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.722039    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94ea251-952e-4a62-bb7d-c39f389542ee-kube-api-access-7px22" pod "e94ea251-952e-4a62-bb7d-c39f389542ee" (UID: "e94ea251-952e-4a62-bb7d-c39f389542ee"). InnerVolumeSpecName "kube-api-access-7px22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:56.783965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.783925    4282 generic.go:356] "Generic (PLEG): container finished" podID="e94ea251-952e-4a62-bb7d-c39f389542ee" containerID="5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d" exitCode=0
Oct 30 22:32:56.784145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.783974    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"
Oct 30 22:32:56.784145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.783999    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv" event={"ID":"e94ea251-952e-4a62-bb7d-c39f389542ee","Type":"ContainerDied","Data":"5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d"}
Oct 30 22:32:56.784145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.784057    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv" event={"ID":"e94ea251-952e-4a62-bb7d-c39f389542ee","Type":"ContainerDied","Data":"54e63c08b06015787558cd9ecb8c5135de7f8d989e0662d79da4cd861eb572c7"}
Oct 30 22:32:56.784145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.784080    4282 scope.go:122] "RemoveContainer" containerID="5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d"
Oct 30 22:32:56.794915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.794897    4282 scope.go:122] "RemoveContainer" containerID="5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d"
Oct 30 22:32:56.795174 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:32:56.795153    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d\": container with ID starting with 5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d not found: ID does not exist" containerID="5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d"
Oct 30 22:32:56.795219 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.795181    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d"} err="failed to get container status \"5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d\": rpc error: code = NotFound desc = could not find container \"5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d\": container with ID starting with 5772a168fda8e4103b79e86989a28843fd509708efe108b580391808182be96d not found: ID does not exist"
Oct 30 22:32:56.807674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.807646    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"]
Oct 30 22:32:56.812806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.812785    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-28b4bb-7c476f8df5-bp6lv"]
Oct 30 22:32:56.821732 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:56.821711    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7px22\" (UniqueName: \"kubernetes.io/projected/e94ea251-952e-4a62-bb7d-c39f389542ee-kube-api-access-7px22\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:32:57.381552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:57.381521    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:32:57.464017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:32:57.463985    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94ea251-952e-4a62-bb7d-c39f389542ee" path="/var/lib/kubelet/pods/e94ea251-952e-4a62-bb7d-c39f389542ee/volumes"
Oct 30 22:33:00.036318 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.036291    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:33:00.062606 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.062580    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:33:00.188847 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.188813    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0438efc5-012a-4fb4-98cf-7b8b9af86145-kube-api-access-rpcjr\" (UniqueName: \"kubernetes.io/projected/0438efc5-012a-4fb4-98cf-7b8b9af86145-kube-api-access-rpcjr\") pod \"0438efc5-012a-4fb4-98cf-7b8b9af86145\" (UID: \"0438efc5-012a-4fb4-98cf-7b8b9af86145\") "
Oct 30 22:33:00.189040 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.188916    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0438efc5-012a-4fb4-98cf-7b8b9af86145-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438efc5-012a-4fb4-98cf-7b8b9af86145-plugins-config-volume\") pod \"0438efc5-012a-4fb4-98cf-7b8b9af86145\" (UID: \"0438efc5-012a-4fb4-98cf-7b8b9af86145\") "
Oct 30 22:33:00.189040 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.188939    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2486a5ab-245c-40cb-93d0-c01ab6408012-kube-api-access-pv5fg\" (UniqueName: \"kubernetes.io/projected/2486a5ab-245c-40cb-93d0-c01ab6408012-kube-api-access-pv5fg\") pod \"2486a5ab-245c-40cb-93d0-c01ab6408012\" (UID: \"2486a5ab-245c-40cb-93d0-c01ab6408012\") "
Oct 30 22:33:00.189040 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.188958    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2486a5ab-245c-40cb-93d0-c01ab6408012-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2486a5ab-245c-40cb-93d0-c01ab6408012-plugins-config-volume\") pod \"2486a5ab-245c-40cb-93d0-c01ab6408012\" (UID: \"2486a5ab-245c-40cb-93d0-c01ab6408012\") "
Oct 30 22:33:00.189399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.189372    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0438efc5-012a-4fb4-98cf-7b8b9af86145-plugins-config-volume" pod "0438efc5-012a-4fb4-98cf-7b8b9af86145" (UID: "0438efc5-012a-4fb4-98cf-7b8b9af86145"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:00.189522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.189372    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2486a5ab-245c-40cb-93d0-c01ab6408012-plugins-config-volume" pod "2486a5ab-245c-40cb-93d0-c01ab6408012" (UID: "2486a5ab-245c-40cb-93d0-c01ab6408012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:00.189635 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.189610    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0438efc5-012a-4fb4-98cf-7b8b9af86145-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.189635 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.189630    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2486a5ab-245c-40cb-93d0-c01ab6408012-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.191232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.191208    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0438efc5-012a-4fb4-98cf-7b8b9af86145-kube-api-access-rpcjr" pod "0438efc5-012a-4fb4-98cf-7b8b9af86145" (UID: "0438efc5-012a-4fb4-98cf-7b8b9af86145"). InnerVolumeSpecName "kube-api-access-rpcjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:00.191308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.191286    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2486a5ab-245c-40cb-93d0-c01ab6408012-kube-api-access-pv5fg" pod "2486a5ab-245c-40cb-93d0-c01ab6408012" (UID: "2486a5ab-245c-40cb-93d0-c01ab6408012"). InnerVolumeSpecName "kube-api-access-pv5fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:00.291825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.291697    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpcjr\" (UniqueName: \"kubernetes.io/projected/0438efc5-012a-4fb4-98cf-7b8b9af86145-kube-api-access-rpcjr\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.291825 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.291732    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv5fg\" (UniqueName: \"kubernetes.io/projected/2486a5ab-245c-40cb-93d0-c01ab6408012-kube-api-access-pv5fg\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:00.812755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.812723    4282 generic.go:356] "Generic (PLEG): container finished" podID="0438efc5-012a-4fb4-98cf-7b8b9af86145" containerID="0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07" exitCode=0
Oct 30 22:33:00.812961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.812767    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"
Oct 30 22:33:00.812961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.812808    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t" event={"ID":"0438efc5-012a-4fb4-98cf-7b8b9af86145","Type":"ContainerDied","Data":"0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07"}
Oct 30 22:33:00.812961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.812873    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t" event={"ID":"0438efc5-012a-4fb4-98cf-7b8b9af86145","Type":"ContainerDied","Data":"fc8ffdba4c0ef270fa7e5af75c72052e3bf8f7949794e623f384252b2b076824"}
Oct 30 22:33:00.812961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.812896    4282 scope.go:122] "RemoveContainer" containerID="0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07"
Oct 30 22:33:00.814615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.814594    4282 generic.go:356] "Generic (PLEG): container finished" podID="2486a5ab-245c-40cb-93d0-c01ab6408012" containerID="756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d" exitCode=0
Oct 30 22:33:00.814739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.814650    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55" event={"ID":"2486a5ab-245c-40cb-93d0-c01ab6408012","Type":"ContainerDied","Data":"756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d"}
Oct 30 22:33:00.814739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.814678    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55" event={"ID":"2486a5ab-245c-40cb-93d0-c01ab6408012","Type":"ContainerDied","Data":"7d505fb789f8e39dd0e0173007cbdb00ec6ad47b3226012f488f4042b482a0ad"}
Oct 30 22:33:00.814824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.814760    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"
Oct 30 22:33:00.832349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.832328    4282 scope.go:122] "RemoveContainer" containerID="0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07"
Oct 30 22:33:00.832857 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:00.832818    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07\": container with ID starting with 0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07 not found: ID does not exist" containerID="0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07"
Oct 30 22:33:00.832988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.832861    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07"} err="failed to get container status \"0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07\": rpc error: code = NotFound desc = could not find container \"0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07\": container with ID starting with 0ffd3f76c1aa2457ddee5b5db75b29d4b564b5643e418206629f153e6452ac07 not found: ID does not exist"
Oct 30 22:33:00.832988 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.832885    4282 scope.go:122] "RemoveContainer" containerID="756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d"
Oct 30 22:33:00.843021 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.842999    4282 scope.go:122] "RemoveContainer" containerID="756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d"
Oct 30 22:33:00.843359 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:00.843337    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d\": container with ID starting with 756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d not found: ID does not exist" containerID="756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d"
Oct 30 22:33:00.843436 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.843366    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d"} err="failed to get container status \"756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d\": rpc error: code = NotFound desc = could not find container \"756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d\": container with ID starting with 756ecc0115c49a7477a8e6139d6f3ecf16f2becffc6fa1d4767d62978d18e40d not found: ID does not exist"
Oct 30 22:33:00.845824 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.845798    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"]
Oct 30 22:33:00.850005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.849983    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a1ea1e-5d8dc7988f-5k22t"]
Oct 30 22:33:00.861449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.861417    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"]
Oct 30 22:33:00.865980 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:00.865959    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a1ea1e-6447659499-jff55"]
Oct 30 22:33:01.465333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:01.465304    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0438efc5-012a-4fb4-98cf-7b8b9af86145" path="/var/lib/kubelet/pods/0438efc5-012a-4fb4-98cf-7b8b9af86145/volumes"
Oct 30 22:33:01.465787 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:01.465773    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2486a5ab-245c-40cb-93d0-c01ab6408012" path="/var/lib/kubelet/pods/2486a5ab-245c-40cb-93d0-c01ab6408012/volumes"
Oct 30 22:33:02.712796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:02.712748    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp" podUID="ae82a63f-9f4c-4a2a-b713-127e52277966" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:33:02.713295 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:02.712763    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs" podUID="0688b402-ba16-4049-a38e-ece12d389c7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:33:03.896851 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:03.896795    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main" containerID="cri-o://6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1" gracePeriod=45
Oct 30 22:33:04.038499 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.038446    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"]
Oct 30 22:33:04.039035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.038970    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7" podUID="caa2f586-028f-4202-8891-5fe93c3650d2" containerName="maas-api" containerID="cri-o://72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2" gracePeriod=30
Oct 30 22:33:04.205784 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.205758    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:33:04.367076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.363953    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-dshm\") pod \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") "
Oct 30 22:33:04.367076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.364756    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-kube-api-access-5t2kl\" (UniqueName: \"kubernetes.io/projected/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-kube-api-access-5t2kl\") pod \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") "
Oct 30 22:33:04.367076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.364869    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tls-certs\") pod \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") "
Oct 30 22:33:04.367076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.364977    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-model-cache\") pod \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") "
Oct 30 22:33:04.367076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.365012    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tmp-dir\") pod \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") "
Oct 30 22:33:04.367076 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.365247    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-home\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-home\") pod \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\" (UID: \"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc\") "
Oct 30 22:33:04.381071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.379841    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-model-cache" pod "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" (UID: "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:04.381071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.380019    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tmp-dir" pod "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" (UID: "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:04.381071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.380535    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.381071 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.380558    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.381729 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.381649    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-home" pod "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" (UID: "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:04.387755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.386684    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tls-certs" pod "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" (UID: "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:04.424048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.414326    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-kube-api-access-5t2kl" pod "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" (UID: "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc"). InnerVolumeSpecName "kube-api-access-5t2kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:04.429662 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.424430    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-dshm" pod "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" (UID: "fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:04.448385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.448359    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:33:04.487527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.485882    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.487527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.485920    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.487527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.485932    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2kl\" (UniqueName: \"kubernetes.io/projected/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-kube-api-access-5t2kl\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.487527 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.485944    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.586642 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.586608    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-metrics-certs\") pod \"caa2f586-028f-4202-8891-5fe93c3650d2\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") "
Oct 30 22:33:04.586850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.586650    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls\") pod \"caa2f586-028f-4202-8891-5fe93c3650d2\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") "
Oct 30 22:33:04.586850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.586758    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa2f586-028f-4202-8891-5fe93c3650d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2f586-028f-4202-8891-5fe93c3650d2-openshift-service-ca\") pod \"caa2f586-028f-4202-8891-5fe93c3650d2\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") "
Oct 30 22:33:04.586850 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.586790    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa2f586-028f-4202-8891-5fe93c3650d2-kube-api-access-2sc7h\" (UniqueName: \"kubernetes.io/projected/caa2f586-028f-4202-8891-5fe93c3650d2-kube-api-access-2sc7h\") pod \"caa2f586-028f-4202-8891-5fe93c3650d2\" (UID: \"caa2f586-028f-4202-8891-5fe93c3650d2\") "
Oct 30 22:33:04.587180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.587154    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa2f586-028f-4202-8891-5fe93c3650d2-openshift-service-ca" pod "caa2f586-028f-4202-8891-5fe93c3650d2" (UID: "caa2f586-028f-4202-8891-5fe93c3650d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:04.587879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.587849    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caa2f586-028f-4202-8891-5fe93c3650d2-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.589008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.588986    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-metrics-certs" pod "caa2f586-028f-4202-8891-5fe93c3650d2" (UID: "caa2f586-028f-4202-8891-5fe93c3650d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:04.589149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.589128    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls" pod "caa2f586-028f-4202-8891-5fe93c3650d2" (UID: "caa2f586-028f-4202-8891-5fe93c3650d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:04.589354 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.589334    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa2f586-028f-4202-8891-5fe93c3650d2-kube-api-access-2sc7h" pod "caa2f586-028f-4202-8891-5fe93c3650d2" (UID: "caa2f586-028f-4202-8891-5fe93c3650d2"). InnerVolumeSpecName "kube-api-access-2sc7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:04.669876 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:04.669839    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:33:04.689228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.689149    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sc7h\" (UniqueName: \"kubernetes.io/projected/caa2f586-028f-4202-8891-5fe93c3650d2-kube-api-access-2sc7h\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.689228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.689179    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.689228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.689189    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caa2f586-028f-4202-8891-5fe93c3650d2-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:04.846485 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.846435    4282 generic.go:356] "Generic (PLEG): container finished" podID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerID="6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1" exitCode=0
Oct 30 22:33:04.846695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.846519    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" event={"ID":"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc","Type":"ContainerDied","Data":"6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1"}
Oct 30 22:33:04.846695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.846537    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"
Oct 30 22:33:04.846695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.846564    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq" event={"ID":"fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc","Type":"ContainerDied","Data":"db8840cfad751a3aa2794d14ebd6b223f314b5768c58df19955ecd0b7b0698cb"}
Oct 30 22:33:04.846695 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.846581    4282 scope.go:122] "RemoveContainer" containerID="6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1"
Oct 30 22:33:04.848457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.848427    4282 generic.go:356] "Generic (PLEG): container finished" podID="caa2f586-028f-4202-8891-5fe93c3650d2" containerID="72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2" exitCode=0
Oct 30 22:33:04.848457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.848477    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7" event={"ID":"caa2f586-028f-4202-8891-5fe93c3650d2","Type":"ContainerDied","Data":"72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2"}
Oct 30 22:33:04.848756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.848499    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"
Oct 30 22:33:04.848756 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.848533    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7" event={"ID":"caa2f586-028f-4202-8891-5fe93c3650d2","Type":"ContainerDied","Data":"38b30aa43b8950669cf47b161290a16de21677955c1a73581c005cebe7daffaf"}
Oct 30 22:33:04.858772 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.858753    4282 scope.go:122] "RemoveContainer" containerID="6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1"
Oct 30 22:33:04.859052 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:04.859028    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1\": container with ID starting with 6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1 not found: ID does not exist" containerID="6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1"
Oct 30 22:33:04.859138 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.859057    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1"} err="failed to get container status \"6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1\": rpc error: code = NotFound desc = could not find container \"6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1\": container with ID starting with 6609b076ee79deb93d1be10a97a71445746fe1bc9df3723d0f0793f8df8119a1 not found: ID does not exist"
Oct 30 22:33:04.859138 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.859072    4282 scope.go:122] "RemoveContainer" containerID="72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2"
Oct 30 22:33:04.870072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.870047    4282 scope.go:122] "RemoveContainer" containerID="72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2"
Oct 30 22:33:04.870593 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:04.870569    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2\": container with ID starting with 72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2 not found: ID does not exist" containerID="72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2"
Oct 30 22:33:04.870751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.870719    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2"} err="failed to get container status \"72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2\": rpc error: code = NotFound desc = could not find container \"72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2\": container with ID starting with 72fd9ef2db3f565963e256b65657b0a4b98cc019fc73f3c089d8f45330597ee2 not found: ID does not exist"
Oct 30 22:33:04.873501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.873482    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"]
Oct 30 22:33:04.877946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.877926    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/simulator-w1-0f26c5-kserve-6c44fb8d46-srxsq"]
Oct 30 22:33:04.888577 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.888558    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"]
Oct 30 22:33:04.895516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:04.895496    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ecfbd6-5b5d86dfdc-frns7"]
Oct 30 22:33:05.466722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:05.466684    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa2f586-028f-4202-8891-5fe93c3650d2" path="/var/lib/kubelet/pods/caa2f586-028f-4202-8891-5fe93c3650d2/volumes"
Oct 30 22:33:05.467199 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:05.467184    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" path="/var/lib/kubelet/pods/fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc/volumes"
Oct 30 22:33:09.704308 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:09.704275    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"]
Oct 30 22:33:09.704804 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:09.704583    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy" containerID="cri-o://3142b42eddc0ad8947f99d5f2f2c915dd010dab40d38d8237fb3efe073db150f" gracePeriod=30
Oct 30 22:33:11.061437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.061394    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"]
Oct 30 22:33:11.061982 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.061639    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7" podUID="21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" containerName="maas-api" containerID="cri-o://f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66" gracePeriod=30
Oct 30 22:33:11.447586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.447560    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 22:33:11.593409 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.593377    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-kube-api-access-xm2q7\" (UniqueName: \"kubernetes.io/projected/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-kube-api-access-xm2q7\") pod \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") "
Oct 30 22:33:11.593623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.593585    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-openshift-service-ca\") pod \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") "
Oct 30 22:33:11.593623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.593620    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-maas-api-tls\") pod \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") "
Oct 30 22:33:11.593727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.593652    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-metrics-certs\") pod \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\" (UID: \"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf\") "
Oct 30 22:33:11.594021 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.593980    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-openshift-service-ca" pod "21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" (UID: "21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:11.594876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.594850    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:11.596079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.596044    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-kube-api-access-xm2q7" pod "21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" (UID: "21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf"). InnerVolumeSpecName "kube-api-access-xm2q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:11.596079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.596065    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-metrics-certs" pod "21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" (UID: "21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:11.596298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.596283    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-maas-api-tls" pod "21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" (UID: "21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:11.696186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.696107    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:11.696186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.696134    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:11.696186 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.696144    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm2q7\" (UniqueName: \"kubernetes.io/projected/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf-kube-api-access-xm2q7\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:11.948764 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.948680    4282 generic.go:356] "Generic (PLEG): container finished" podID="21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" containerID="f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66" exitCode=0
Oct 30 22:33:11.948764 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.948740    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"
Oct 30 22:33:11.948962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.948757    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7" event={"ID":"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf","Type":"ContainerDied","Data":"f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66"}
Oct 30 22:33:11.948962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.948816    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7" event={"ID":"21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf","Type":"ContainerDied","Data":"855c924a3a1a52108e6d2c6dbf2bb856dee02dcc8637ead415f77ce08eca444a"}
Oct 30 22:33:11.948962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.948840    4282 scope.go:122] "RemoveContainer" containerID="f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66"
Oct 30 22:33:11.959572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.959553    4282 scope.go:122] "RemoveContainer" containerID="f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66"
Oct 30 22:33:11.959857 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:11.959832    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66\": container with ID starting with f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66 not found: ID does not exist" containerID="f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66"
Oct 30 22:33:11.959914 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.959869    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66"} err="failed to get container status \"f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66\": rpc error: code = NotFound desc = could not find container \"f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66\": container with ID starting with f3c15ad5eb4f8be2c18e82d6bfb1ebe9d733e41abed80d1349876d8555261e66 not found: ID does not exist"
Oct 30 22:33:11.979271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.979245    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"]
Oct 30 22:33:11.990946 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:11.990925    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-0f26c5-56dcd45879-hwdm7"]
Oct 30 22:33:12.712521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:12.712483    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp" podUID="ae82a63f-9f4c-4a2a-b713-127e52277966" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:33:12.712929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:12.712495    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs" podUID="0688b402-ba16-4049-a38e-ece12d389c7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:33:13.032185 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:13.032106    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"]
Oct 30 22:33:13.032362 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:13.032341    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy" containerID="cri-o://7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e" gracePeriod=30
Oct 30 22:33:13.464176 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:13.464142    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" path="/var/lib/kubelet/pods/21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf/volumes"
Oct 30 22:33:15.010144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.010058    4282 generic.go:356] "Generic (PLEG): container finished" podID="774525b8-c126-4535-a248-0b63d68e599a" containerID="3142b42eddc0ad8947f99d5f2f2c915dd010dab40d38d8237fb3efe073db150f" exitCode=0
Oct 30 22:33:15.010584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.010291    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" event={"ID":"774525b8-c126-4535-a248-0b63d68e599a","Type":"ContainerDied","Data":"3142b42eddc0ad8947f99d5f2f2c915dd010dab40d38d8237fb3efe073db150f"}
Oct 30 22:33:15.010584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.010346    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b" event={"ID":"774525b8-c126-4535-a248-0b63d68e599a","Type":"ContainerDied","Data":"3b4c4f1d0ca1d4d5645cb92c1639c69a7cd5e644baaa5a5b541eff835b88e7f9"}
Oct 30 22:33:15.010584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.010372    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b4c4f1d0ca1d4d5645cb92c1639c69a7cd5e644baaa5a5b541eff835b88e7f9"
Oct 30 22:33:15.010584 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.010388    4282 scope.go:122] "RemoveContainer" containerID="9d893cc82712c7ab4628cbda09cdc6d50e0e1db806d28d87dd88823edab40961"
Oct 30 22:33:15.013930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.013907    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:33:15.141066 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141034    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-credential-socket\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.141266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141120    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-kube-api-access-ckqkg\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-kube-api-access-ckqkg\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.141266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141160    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-socket\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.141266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141200    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-certs\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.141266 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141264    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/774525b8-c126-4535-a248-0b63d68e599a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774525b8-c126-4535-a248-0b63d68e599a-istio-podinfo\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141284    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-envoy\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141341    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-credential-socket" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141358    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-istio-token\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-istio-token\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141435    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774525b8-c126-4535-a248-0b63d68e599a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774525b8-c126-4535-a248-0b63d68e599a-istiod-ca-cert\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141496    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-data\") pod \"774525b8-c126-4535-a248-0b63d68e599a\" (UID: \"774525b8-c126-4535-a248-0b63d68e599a\") "
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141598    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-certs" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.141877    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-data" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.142115    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.142301    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774525b8-c126-4535-a248-0b63d68e599a-istiod-ca-cert" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:15.142568 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.142506    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-socket" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:15.143807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.142201    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:15.143941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.143818    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:15.144048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.144025    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-envoy" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:15.144272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.144242    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/774525b8-c126-4535-a248-0b63d68e599a-istio-podinfo" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:15.144272 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.144254    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-kube-api-access-ckqkg" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "kube-api-access-ckqkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:15.144543 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.144523    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-istio-token" pod "774525b8-c126-4535-a248-0b63d68e599a" (UID: "774525b8-c126-4535-a248-0b63d68e599a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:15.245583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.245548    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/774525b8-c126-4535-a248-0b63d68e599a-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:15.245583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.245579    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:15.245583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.245590    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:15.245805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.245598    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/774525b8-c126-4535-a248-0b63d68e599a-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:15.245805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.245607    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckqkg\" (UniqueName: \"kubernetes.io/projected/774525b8-c126-4535-a248-0b63d68e599a-kube-api-access-ckqkg\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:15.245805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:15.245616    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/774525b8-c126-4535-a248-0b63d68e599a-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:16.018382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.018359    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"
Oct 30 22:33:16.136829 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.136730    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"]
Oct 30 22:33:16.141843 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.141438    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5f192-openshift-default-7d4b6c654f-wfr6b"]
Oct 30 22:33:16.296905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.296876    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:33:16.319428 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.319401    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:33:16.361977 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.361948    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0688b402-ba16-4049-a38e-ece12d389c7d-kube-api-access-qx2n2\" (UniqueName: \"kubernetes.io/projected/0688b402-ba16-4049-a38e-ece12d389c7d-kube-api-access-qx2n2\") pod \"0688b402-ba16-4049-a38e-ece12d389c7d\" (UID: \"0688b402-ba16-4049-a38e-ece12d389c7d\") "
Oct 30 22:33:16.362175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.362162    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0688b402-ba16-4049-a38e-ece12d389c7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0688b402-ba16-4049-a38e-ece12d389c7d-plugins-config-volume\") pod \"0688b402-ba16-4049-a38e-ece12d389c7d\" (UID: \"0688b402-ba16-4049-a38e-ece12d389c7d\") "
Oct 30 22:33:16.362528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.362506    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0688b402-ba16-4049-a38e-ece12d389c7d-plugins-config-volume" pod "0688b402-ba16-4049-a38e-ece12d389c7d" (UID: "0688b402-ba16-4049-a38e-ece12d389c7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:16.364170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.364152    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0688b402-ba16-4049-a38e-ece12d389c7d-kube-api-access-qx2n2" pod "0688b402-ba16-4049-a38e-ece12d389c7d" (UID: "0688b402-ba16-4049-a38e-ece12d389c7d"). InnerVolumeSpecName "kube-api-access-qx2n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:16.463983 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.463953    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae82a63f-9f4c-4a2a-b713-127e52277966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae82a63f-9f4c-4a2a-b713-127e52277966-plugins-config-volume\") pod \"ae82a63f-9f4c-4a2a-b713-127e52277966\" (UID: \"ae82a63f-9f4c-4a2a-b713-127e52277966\") "
Oct 30 22:33:16.464192 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.464000    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae82a63f-9f4c-4a2a-b713-127e52277966-kube-api-access-rqqlj\" (UniqueName: \"kubernetes.io/projected/ae82a63f-9f4c-4a2a-b713-127e52277966-kube-api-access-rqqlj\") pod \"ae82a63f-9f4c-4a2a-b713-127e52277966\" (UID: \"ae82a63f-9f4c-4a2a-b713-127e52277966\") "
Oct 30 22:33:16.464414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.464388    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae82a63f-9f4c-4a2a-b713-127e52277966-plugins-config-volume" pod "ae82a63f-9f4c-4a2a-b713-127e52277966" (UID: "ae82a63f-9f4c-4a2a-b713-127e52277966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:16.464525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.464505    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx2n2\" (UniqueName: \"kubernetes.io/projected/0688b402-ba16-4049-a38e-ece12d389c7d-kube-api-access-qx2n2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:16.464525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.464524    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0688b402-ba16-4049-a38e-ece12d389c7d-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:16.466284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.466267    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae82a63f-9f4c-4a2a-b713-127e52277966-kube-api-access-rqqlj" pod "ae82a63f-9f4c-4a2a-b713-127e52277966" (UID: "ae82a63f-9f4c-4a2a-b713-127e52277966"). InnerVolumeSpecName "kube-api-access-rqqlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:16.565502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.565393    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae82a63f-9f4c-4a2a-b713-127e52277966-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:16.565502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:16.565426    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqqlj\" (UniqueName: \"kubernetes.io/projected/ae82a63f-9f4c-4a2a-b713-127e52277966-kube-api-access-rqqlj\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:17.026941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.026908    4282 generic.go:356] "Generic (PLEG): container finished" podID="ae82a63f-9f4c-4a2a-b713-127e52277966" containerID="6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4" exitCode=0
Oct 30 22:33:17.027413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.026958    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"
Oct 30 22:33:17.027413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.026995    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp" event={"ID":"ae82a63f-9f4c-4a2a-b713-127e52277966","Type":"ContainerDied","Data":"6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4"}
Oct 30 22:33:17.027413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.027063    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp" event={"ID":"ae82a63f-9f4c-4a2a-b713-127e52277966","Type":"ContainerDied","Data":"0994c659ca1245f2a79e847bc35dd865c28f47b56cd68f6edced6350447040dc"}
Oct 30 22:33:17.027413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.027079    4282 scope.go:122] "RemoveContainer" containerID="6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4"
Oct 30 22:33:17.028615 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.028593    4282 generic.go:356] "Generic (PLEG): container finished" podID="0688b402-ba16-4049-a38e-ece12d389c7d" containerID="86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03" exitCode=0
Oct 30 22:33:17.028730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.028621    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs" event={"ID":"0688b402-ba16-4049-a38e-ece12d389c7d","Type":"ContainerDied","Data":"86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03"}
Oct 30 22:33:17.028730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.028630    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"
Oct 30 22:33:17.028730 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.028654    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs" event={"ID":"0688b402-ba16-4049-a38e-ece12d389c7d","Type":"ContainerDied","Data":"24f118d0b62df7631104226a993b230342eca235f5f4363f5c64aaeb6f92c5fa"}
Oct 30 22:33:17.038631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.038611    4282 scope.go:122] "RemoveContainer" containerID="6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4"
Oct 30 22:33:17.038917 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:17.038896    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4\": container with ID starting with 6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4 not found: ID does not exist" containerID="6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4"
Oct 30 22:33:17.039093 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.038923    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4"} err="failed to get container status \"6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4\": rpc error: code = NotFound desc = could not find container \"6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4\": container with ID starting with 6e732e204f18881e1eca1c1432c9f8745864a542e68792612c86e73ffff39dc4 not found: ID does not exist"
Oct 30 22:33:17.039093 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.038940    4282 scope.go:122] "RemoveContainer" containerID="86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03"
Oct 30 22:33:17.047933 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.047915    4282 scope.go:122] "RemoveContainer" containerID="86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03"
Oct 30 22:33:17.048182 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:17.048162    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03\": container with ID starting with 86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03 not found: ID does not exist" containerID="86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03"
Oct 30 22:33:17.048232 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.048190    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03"} err="failed to get container status \"86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03\": rpc error: code = NotFound desc = could not find container \"86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03\": container with ID starting with 86c5754d0b1f7572eb9369355b3fb7b8ee646368c33183e7b54fa4be6223ed03 not found: ID does not exist"
Oct 30 22:33:17.053728 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.053706    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"]
Oct 30 22:33:17.057636 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.057616    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a5f192-f4dc7c4c-99psp"]
Oct 30 22:33:17.067723 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.067704    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"]
Oct 30 22:33:17.073327 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.073308    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5f192-dfc895cfc-bqkzs"]
Oct 30 22:33:17.463404 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.463368    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0688b402-ba16-4049-a38e-ece12d389c7d" path="/var/lib/kubelet/pods/0688b402-ba16-4049-a38e-ece12d389c7d/volumes"
Oct 30 22:33:17.463916 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.463901    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774525b8-c126-4535-a248-0b63d68e599a" path="/var/lib/kubelet/pods/774525b8-c126-4535-a248-0b63d68e599a/volumes"
Oct 30 22:33:17.464282 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:17.464270    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae82a63f-9f4c-4a2a-b713-127e52277966" path="/var/lib/kubelet/pods/ae82a63f-9f4c-4a2a-b713-127e52277966/volumes"
Oct 30 22:33:18.312864 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.312805    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:33:18.387871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.387840    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istiod-ca-cert\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.387871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.387876    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-podinfo\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.388120 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.387935    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-credential-socket\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.388120 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.387955    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-kube-api-access-cwc7g\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-kube-api-access-cwc7g\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.388225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388135    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-envoy\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.388276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388224    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-certs\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.388276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388256    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-credential-socket" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.388394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388299    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-socket\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.388394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388302    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istiod-ca-cert" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:18.388394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388365    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-data\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.388592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388544    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-certs" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.388652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388584    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-socket" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.388652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388621    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-token\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-token\") pod \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\" (UID: \"f3b514a9-b9c4-4705-a789-a7acaba6a27f\") "
Oct 30 22:33:18.388781 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.388758    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-data" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.389410 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.389374    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.389410 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.389408    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.389585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.389426    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.389585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.389442    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.389585 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.389457    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.390628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.390588    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-envoy" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:18.390874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.390850    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-podinfo" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:18.390874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.390859    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-kube-api-access-cwc7g" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "kube-api-access-cwc7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:18.391088 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.391069    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-token" pod "f3b514a9-b9c4-4705-a789-a7acaba6a27f" (UID: "f3b514a9-b9c4-4705-a789-a7acaba6a27f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:18.505697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.505607    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.505697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.505641    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.505697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.505654    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwc7g\" (UniqueName: \"kubernetes.io/projected/f3b514a9-b9c4-4705-a789-a7acaba6a27f-kube-api-access-cwc7g\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.505697 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.505670    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b514a9-b9c4-4705-a789-a7acaba6a27f-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:18.750015 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.749982    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"]
Oct 30 22:33:18.750246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:18.750217    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy" containerID="cri-o://e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1" gracePeriod=30
Oct 30 22:33:19.054256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.054174    4282 generic.go:356] "Generic (PLEG): container finished" podID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerID="7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e" exitCode=0
Oct 30 22:33:19.054256 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.054232    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"
Oct 30 22:33:19.054516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.054254    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" event={"ID":"f3b514a9-b9c4-4705-a789-a7acaba6a27f","Type":"ContainerDied","Data":"7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e"}
Oct 30 22:33:19.054516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.054311    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm" event={"ID":"f3b514a9-b9c4-4705-a789-a7acaba6a27f","Type":"ContainerDied","Data":"c776b3de776771bdbc6bb886ffed5540e623fde9689bf941f15b26083ab5916e"}
Oct 30 22:33:19.054516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.054334    4282 scope.go:122] "RemoveContainer" containerID="7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e"
Oct 30 22:33:19.066342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.066325    4282 scope.go:122] "RemoveContainer" containerID="df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d"
Oct 30 22:33:19.075817 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.075798    4282 scope.go:122] "RemoveContainer" containerID="7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e"
Oct 30 22:33:19.076111 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:19.076092    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e\": container with ID starting with 7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e not found: ID does not exist" containerID="7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e"
Oct 30 22:33:19.076170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.076118    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e"} err="failed to get container status \"7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e\": rpc error: code = NotFound desc = could not find container \"7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e\": container with ID starting with 7ebe72ddcecee8e9d7f097303361e2a7b509468aa5a81713780eff5be603a67e not found: ID does not exist"
Oct 30 22:33:19.076170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.076138    4282 scope.go:122] "RemoveContainer" containerID="df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d"
Oct 30 22:33:19.076403 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:19.076382    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d\": container with ID starting with df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d not found: ID does not exist" containerID="df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d"
Oct 30 22:33:19.076457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.076409    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d"} err="failed to get container status \"df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d\": rpc error: code = NotFound desc = could not find container \"df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d\": container with ID starting with df3e8b0e8c2eea4310bb49d8fd42e0ab05d3c73dfd61c49944083628b268b46d not found: ID does not exist"
Oct 30 22:33:19.077706 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.077681    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"]
Oct 30 22:33:19.082217 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.082195    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ecfbd6-openshift-default-7cd598dd79-95lxm"]
Oct 30 22:33:19.463176 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:19.463144    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" path="/var/lib/kubelet/pods/f3b514a9-b9c4-4705-a789-a7acaba6a27f/volumes"
Oct 30 22:33:20.346649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:20.346613    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"]
Oct 30 22:33:22.833143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:22.833111    4282 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:33:22.833594 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:22.833160    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:33:23.799586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:23.799554    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"]
Oct 30 22:33:24.149519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.143092    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 22:33:24.149519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.148164    4282 generic.go:356] "Generic (PLEG): container finished" podID="f6d46316-4736-474f-aa55-71da33c19818" containerID="e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1" exitCode=0
Oct 30 22:33:24.149519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.148414    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" event={"ID":"f6d46316-4736-474f-aa55-71da33c19818","Type":"ContainerDied","Data":"e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1"}
Oct 30 22:33:24.149519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.148521    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8" event={"ID":"f6d46316-4736-474f-aa55-71da33c19818","Type":"ContainerDied","Data":"8466c75e4a4a449f989552dc33344dcb0a41ad8aaf43d3c00f003c00399f92b9"}
Oct 30 22:33:24.149519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.148557    4282 scope.go:122] "RemoveContainer" containerID="e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1"
Oct 30 22:33:24.149519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.148706    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"
Oct 30 22:33:24.163268 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.163250    4282 scope.go:122] "RemoveContainer" containerID="e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1"
Oct 30 22:33:24.163643 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:24.163603    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1\": container with ID starting with e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1 not found: ID does not exist" containerID="e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1"
Oct 30 22:33:24.163717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.163646    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1"} err="failed to get container status \"e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1\": rpc error: code = NotFound desc = could not find container \"e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1\": container with ID starting with e56fb4e816c449677e362a9fb3276f354bd8c5dc86ad19e07be96fa5a314c4d1 not found: ID does not exist"
Oct 30 22:33:24.230941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.230909    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-data\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.230947    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-kube-api-access-7zz64\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-kube-api-access-7zz64\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231130 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.230981    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-certs\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231120    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-istio-token\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-istio-token\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231171    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-credential-socket\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231215    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-socket\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231279    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-data" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:24.231426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231303    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6d46316-4736-474f-aa55-71da33c19818-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d46316-4736-474f-aa55-71da33c19818-istio-podinfo\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231351    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-envoy\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231426 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231382    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-certs" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:24.231679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231446    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6d46316-4736-474f-aa55-71da33c19818-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d46316-4736-474f-aa55-71da33c19818-istiod-ca-cert\") pod \"f6d46316-4736-474f-aa55-71da33c19818\" (UID: \"f6d46316-4736-474f-aa55-71da33c19818\") "
Oct 30 22:33:24.231679 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.231546    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-socket" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:24.232164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.232035    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.232296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.232179    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.232296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.232198    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.232510 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.232358    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-credential-socket" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:24.232612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.232574    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6d46316-4736-474f-aa55-71da33c19818-istiod-ca-cert" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:24.233838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.233812    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-istio-token" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:24.234112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.234089    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6d46316-4736-474f-aa55-71da33c19818-istio-podinfo" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:24.234237 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.234204    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-envoy" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:24.235004 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.234913    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-kube-api-access-7zz64" pod "f6d46316-4736-474f-aa55-71da33c19818" (UID: "f6d46316-4736-474f-aa55-71da33c19818"). InnerVolumeSpecName "kube-api-access-7zz64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:24.333849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.333776    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6d46316-4736-474f-aa55-71da33c19818-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.333849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.333803    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zz64\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-kube-api-access-7zz64\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.333849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.333813    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6d46316-4736-474f-aa55-71da33c19818-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.333849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.333823    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.333849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.333833    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6d46316-4736-474f-aa55-71da33c19818-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.333849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.333842    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6d46316-4736-474f-aa55-71da33c19818-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:24.475750 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.475723    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"]
Oct 30 22:33:24.480193 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:24.480171    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-7dxh8"]
Oct 30 22:33:25.347146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.347103    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w" podUID="9fd149ee-426a-4670-b647-02f50aef6219" containerName="payload-pre-processing" containerID="cri-o://278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414" gracePeriod=25
Oct 30 22:33:25.464880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.464851    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6d46316-4736-474f-aa55-71da33c19818" path="/var/lib/kubelet/pods/f6d46316-4736-474f-aa55-71da33c19818/volumes"
Oct 30 22:33:25.626074 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.626049    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:33:25.648439 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.648417    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fd149ee-426a-4670-b647-02f50aef6219-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd149ee-426a-4670-b647-02f50aef6219-plugins-config-volume\") pod \"9fd149ee-426a-4670-b647-02f50aef6219\" (UID: \"9fd149ee-426a-4670-b647-02f50aef6219\") "
Oct 30 22:33:25.648728 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.648707    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fd149ee-426a-4670-b647-02f50aef6219-kube-api-access-hqhz2\" (UniqueName: \"kubernetes.io/projected/9fd149ee-426a-4670-b647-02f50aef6219-kube-api-access-hqhz2\") pod \"9fd149ee-426a-4670-b647-02f50aef6219\" (UID: \"9fd149ee-426a-4670-b647-02f50aef6219\") "
Oct 30 22:33:25.650501 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.648818    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd149ee-426a-4670-b647-02f50aef6219-plugins-config-volume" pod "9fd149ee-426a-4670-b647-02f50aef6219" (UID: "9fd149ee-426a-4670-b647-02f50aef6219"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:25.651146 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.651123    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd149ee-426a-4670-b647-02f50aef6219-kube-api-access-hqhz2" pod "9fd149ee-426a-4670-b647-02f50aef6219" (UID: "9fd149ee-426a-4670-b647-02f50aef6219"). InnerVolumeSpecName "kube-api-access-hqhz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:25.750126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.750095    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd149ee-426a-4670-b647-02f50aef6219-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:25.750126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:25.750121    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqhz2\" (UniqueName: \"kubernetes.io/projected/9fd149ee-426a-4670-b647-02f50aef6219-kube-api-access-hqhz2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:26.163589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.163553    4282 generic.go:356] "Generic (PLEG): container finished" podID="9fd149ee-426a-4670-b647-02f50aef6219" containerID="278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414" exitCode=0
Oct 30 22:33:26.163805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.163603    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"
Oct 30 22:33:26.163805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.163643    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w" event={"ID":"9fd149ee-426a-4670-b647-02f50aef6219","Type":"ContainerDied","Data":"278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414"}
Oct 30 22:33:26.163805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.163707    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w" event={"ID":"9fd149ee-426a-4670-b647-02f50aef6219","Type":"ContainerDied","Data":"43abb7a8560b0d5ce7c8e6ca61927703d302a8abdc2411dcdca7aca10f7de917"}
Oct 30 22:33:26.163805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.163737    4282 scope.go:122] "RemoveContainer" containerID="278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414"
Oct 30 22:33:26.173645 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.173626    4282 scope.go:122] "RemoveContainer" containerID="278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414"
Oct 30 22:33:26.173927 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:26.173907    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414\": container with ID starting with 278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414 not found: ID does not exist" containerID="278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414"
Oct 30 22:33:26.173980 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.173935    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414"} err="failed to get container status \"278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414\": rpc error: code = NotFound desc = could not find container \"278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414\": container with ID starting with 278061d72bf6858dab0a7be0d1e4554dd583a8b758ecd96113b1bd74c3cfd414 not found: ID does not exist"
Oct 30 22:33:26.185727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.185705    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"]
Oct 30 22:33:26.189790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:26.189767    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cca1c7-65b5c99747-qkm7w"]
Oct 30 22:33:27.470497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:27.469784    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fd149ee-426a-4670-b647-02f50aef6219" path="/var/lib/kubelet/pods/9fd149ee-426a-4670-b647-02f50aef6219/volumes"
Oct 30 22:33:28.800562 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:28.800509    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn" podUID="cc889df4-c49f-4b23-9db5-2561028d5b2e" containerName="payload-pre-processing" containerID="cri-o://6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836" gracePeriod=25
Oct 30 22:33:30.041569 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.041533    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"]
Oct 30 22:33:30.042135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.041802    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497" podUID="7760b38b-dcfa-463f-969a-954a60353919" containerName="maas-api" containerID="cri-o://09637470fd959ebd88e691b551ba4bfbb758f0fb29190ad722362dcb60b55bd5" gracePeriod=30
Oct 30 22:33:30.192188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.192158    4282 generic.go:356] "Generic (PLEG): container finished" podID="7760b38b-dcfa-463f-969a-954a60353919" containerID="09637470fd959ebd88e691b551ba4bfbb758f0fb29190ad722362dcb60b55bd5" exitCode=0
Oct 30 22:33:30.192326 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.192226    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497" event={"ID":"7760b38b-dcfa-463f-969a-954a60353919","Type":"ContainerDied","Data":"09637470fd959ebd88e691b551ba4bfbb758f0fb29190ad722362dcb60b55bd5"}
Oct 30 22:33:30.399674 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.399642    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:33:30.406837 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.406811    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7760b38b-dcfa-463f-969a-954a60353919-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7760b38b-dcfa-463f-969a-954a60353919-openshift-service-ca\") pod \"7760b38b-dcfa-463f-969a-954a60353919\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") "
Oct 30 22:33:30.407002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.406927    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-metrics-certs\") pod \"7760b38b-dcfa-463f-969a-954a60353919\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") "
Oct 30 22:33:30.407002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.406970    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7760b38b-dcfa-463f-969a-954a60353919-kube-api-access-wrhhq\" (UniqueName: \"kubernetes.io/projected/7760b38b-dcfa-463f-969a-954a60353919-kube-api-access-wrhhq\") pod \"7760b38b-dcfa-463f-969a-954a60353919\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") "
Oct 30 22:33:30.407140 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.407013    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls\") pod \"7760b38b-dcfa-463f-969a-954a60353919\" (UID: \"7760b38b-dcfa-463f-969a-954a60353919\") "
Oct 30 22:33:30.407230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.407203    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7760b38b-dcfa-463f-969a-954a60353919-openshift-service-ca" pod "7760b38b-dcfa-463f-969a-954a60353919" (UID: "7760b38b-dcfa-463f-969a-954a60353919"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:30.407659 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.407588    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7760b38b-dcfa-463f-969a-954a60353919-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.409547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.409520    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls" pod "7760b38b-dcfa-463f-969a-954a60353919" (UID: "7760b38b-dcfa-463f-969a-954a60353919"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:30.409739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.409724    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-metrics-certs" pod "7760b38b-dcfa-463f-969a-954a60353919" (UID: "7760b38b-dcfa-463f-969a-954a60353919"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:30.409819 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.409776    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7760b38b-dcfa-463f-969a-954a60353919-kube-api-access-wrhhq" pod "7760b38b-dcfa-463f-969a-954a60353919" (UID: "7760b38b-dcfa-463f-969a-954a60353919"). InnerVolumeSpecName "kube-api-access-wrhhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:30.509257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.509222    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.509257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.509252    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrhhq\" (UniqueName: \"kubernetes.io/projected/7760b38b-dcfa-463f-969a-954a60353919-kube-api-access-wrhhq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:30.509257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:30.509262    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7760b38b-dcfa-463f-969a-954a60353919-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:31.201941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:31.201913    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"
Oct 30 22:33:31.201941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:31.201931    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497" event={"ID":"7760b38b-dcfa-463f-969a-954a60353919","Type":"ContainerDied","Data":"801437547a534fb98d46478d11a7b7ae81d468b71042e9294d641b9a911a02de"}
Oct 30 22:33:31.202506 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:31.202005    4282 scope.go:122] "RemoveContainer" containerID="09637470fd959ebd88e691b551ba4bfbb758f0fb29190ad722362dcb60b55bd5"
Oct 30 22:33:31.291760 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:31.291729    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"]
Oct 30 22:33:31.296495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:31.296445    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cca1c7-6f8764684d-w6497"]
Oct 30 22:33:31.465231 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:31.465148    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7760b38b-dcfa-463f-969a-954a60353919" path="/var/lib/kubelet/pods/7760b38b-dcfa-463f-969a-954a60353919/volumes"
Oct 30 22:33:34.632810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:34.632776    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn" podUID="cc889df4-c49f-4b23-9db5-2561028d5b2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:33:34.649249 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:34.649219    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:33:37.528476 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:37.528422    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:33:39.465752 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:39.465714    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"]
Oct 30 22:33:39.466262 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:39.465989    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy" containerID="cri-o://07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7" gracePeriod=30
Oct 30 22:33:44.632592 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.632551    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn" podUID="cc889df4-c49f-4b23-9db5-2561028d5b2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:33:44.756724 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.756690    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:33:44.853279 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853250    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-kube-api-access-c47pf\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-kube-api-access-c47pf\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853303    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-token\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-token\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853338    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-certs\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853379    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-data\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853419    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-credential-socket\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853733 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853454    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-socket\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853733 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853646    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-envoy\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853733 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853698    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-podinfo\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853766    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istiod-ca-cert\") pod \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\" (UID: \"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c\") "
Oct 30 22:33:44.853906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853857    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-socket" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:44.853906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.853868    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-credential-socket" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:44.854251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.854228    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istiod-ca-cert" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:44.854347 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.854292    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.854347 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.854307    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.854347 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.854304    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-certs" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:44.854652 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.854631    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-data" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:44.856883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.856856    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-kube-api-access-c47pf" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "kube-api-access-c47pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:44.856883 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.856868    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-token" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:44.857023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.856902    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-podinfo" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:44.857023 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.856904    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-envoy" pod "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" (UID: "d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:44.955288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.955252    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.955288 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.955288    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.955554 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.955304    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.955554 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.955319    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.955554 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.955333    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.955554 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.955347    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.955554 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.955365    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c47pf\" (UniqueName: \"kubernetes.io/projected/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c-kube-api-access-c47pf\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:44.965271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.965233    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"]
Oct 30 22:33:44.965599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.965573    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy" containerID="cri-o://7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b" gracePeriod=30
Oct 30 22:33:44.976646 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:44.976616    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"]
Oct 30 22:33:45.000249 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.000183    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"]
Oct 30 22:33:45.312700 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.312613    4282 generic.go:356] "Generic (PLEG): container finished" podID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerID="07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7" exitCode=0
Oct 30 22:33:45.312700 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.312669    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"
Oct 30 22:33:45.312915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.312693    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" event={"ID":"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c","Type":"ContainerDied","Data":"07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7"}
Oct 30 22:33:45.312915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.312752    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8" event={"ID":"d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c","Type":"ContainerDied","Data":"6d93d9c4482f816c4cea24e41b2480412c7e605fbc7fa882240a25fd09fb0a4c"}
Oct 30 22:33:45.312915 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.312776    4282 scope.go:122] "RemoveContainer" containerID="07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7"
Oct 30 22:33:45.323491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.323442    4282 scope.go:122] "RemoveContainer" containerID="7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da"
Oct 30 22:33:45.333059 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.333041    4282 scope.go:122] "RemoveContainer" containerID="07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7"
Oct 30 22:33:45.333356 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:45.333334    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7\": container with ID starting with 07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7 not found: ID does not exist" containerID="07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7"
Oct 30 22:33:45.333457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.333362    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7"} err="failed to get container status \"07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7\": rpc error: code = NotFound desc = could not find container \"07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7\": container with ID starting with 07f9e70cf9107f71e288f3b11a778d935516a9d33097e8a8d69fe91fcc14fea7 not found: ID does not exist"
Oct 30 22:33:45.333457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.333382    4282 scope.go:122] "RemoveContainer" containerID="7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da"
Oct 30 22:33:45.333740 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:45.333720    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da\": container with ID starting with 7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da not found: ID does not exist" containerID="7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da"
Oct 30 22:33:45.333796 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.333747    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da"} err="failed to get container status \"7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da\": rpc error: code = NotFound desc = could not find container \"7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da\": container with ID starting with 7fd8b1f760b915136cb947f9dd54a737c9c8c7c1156bbf930e24de27b5fd94da not found: ID does not exist"
Oct 30 22:33:45.335061 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.335043    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"]
Oct 30 22:33:45.341895 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.341876    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0a2577-openshift-default-5dff5cb66f-mvlx8"]
Oct 30 22:33:45.464572 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:45.464544    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" path="/var/lib/kubelet/pods/d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c/volumes"
Oct 30 22:33:46.172246 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:46.172218    4282 scope.go:122] "RemoveContainer" containerID="efd2e4ae86e0db3cf23d0bfb8c41f65653965a3a13d0afdaf95d5ecdefb39c83"
Oct 30 22:33:46.182271 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:46.182253    4282 scope.go:122] "RemoveContainer" containerID="3142b42eddc0ad8947f99d5f2f2c915dd010dab40d38d8237fb3efe073db150f"
Oct 30 22:33:46.191389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:46.191361    4282 scope.go:122] "RemoveContainer" containerID="fcd8b766b5ce9cb275773b409311c243f507e96fe564d649043aa8dae887596f"
Oct 30 22:33:48.368583 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:48.368551    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"]
Oct 30 22:33:49.072406 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.072351    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:33:49.153270 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.153244    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc889df4-c49f-4b23-9db5-2561028d5b2e-kube-api-access-jwpvs\" (UniqueName: \"kubernetes.io/projected/cc889df4-c49f-4b23-9db5-2561028d5b2e-kube-api-access-jwpvs\") pod \"cc889df4-c49f-4b23-9db5-2561028d5b2e\" (UID: \"cc889df4-c49f-4b23-9db5-2561028d5b2e\") "
Oct 30 22:33:49.153431 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.153327    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc889df4-c49f-4b23-9db5-2561028d5b2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc889df4-c49f-4b23-9db5-2561028d5b2e-plugins-config-volume\") pod \"cc889df4-c49f-4b23-9db5-2561028d5b2e\" (UID: \"cc889df4-c49f-4b23-9db5-2561028d5b2e\") "
Oct 30 22:33:49.153744 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.153719    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc889df4-c49f-4b23-9db5-2561028d5b2e-plugins-config-volume" pod "cc889df4-c49f-4b23-9db5-2561028d5b2e" (UID: "cc889df4-c49f-4b23-9db5-2561028d5b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:49.155693 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.155665    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc889df4-c49f-4b23-9db5-2561028d5b2e-kube-api-access-jwpvs" pod "cc889df4-c49f-4b23-9db5-2561028d5b2e" (UID: "cc889df4-c49f-4b23-9db5-2561028d5b2e"). InnerVolumeSpecName "kube-api-access-jwpvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:49.254835 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.254802    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwpvs\" (UniqueName: \"kubernetes.io/projected/cc889df4-c49f-4b23-9db5-2561028d5b2e-kube-api-access-jwpvs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:49.254835 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.254830    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc889df4-c49f-4b23-9db5-2561028d5b2e-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:49.340961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.340865    4282 generic.go:356] "Generic (PLEG): container finished" podID="cc889df4-c49f-4b23-9db5-2561028d5b2e" containerID="6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836" exitCode=0
Oct 30 22:33:49.340961 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.340917    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"
Oct 30 22:33:49.341233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.340957    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn" event={"ID":"cc889df4-c49f-4b23-9db5-2561028d5b2e","Type":"ContainerDied","Data":"6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836"}
Oct 30 22:33:49.341233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.341011    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn" event={"ID":"cc889df4-c49f-4b23-9db5-2561028d5b2e","Type":"ContainerDied","Data":"85909a548cd0efcf8ee5feb45791383a7ae5e2fc39ed486253be91574e26da07"}
Oct 30 22:33:49.341233 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.341024    4282 scope.go:122] "RemoveContainer" containerID="6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836"
Oct 30 22:33:49.355166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.355146    4282 scope.go:122] "RemoveContainer" containerID="6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836"
Oct 30 22:33:49.355532 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:49.355512    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836\": container with ID starting with 6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836 not found: ID does not exist" containerID="6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836"
Oct 30 22:33:49.355619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.355539    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836"} err="failed to get container status \"6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836\": rpc error: code = NotFound desc = could not find container \"6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836\": container with ID starting with 6ca30de87fe0da848dd3f3709ad07e804d80cc9f390e77df5492a587e1549836 not found: ID does not exist"
Oct 30 22:33:49.364597 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.364574    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"]
Oct 30 22:33:49.370970 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.370951    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-0a2577-8485c7987-4sfcn"]
Oct 30 22:33:49.463207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:49.463178    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc889df4-c49f-4b23-9db5-2561028d5b2e" path="/var/lib/kubelet/pods/cc889df4-c49f-4b23-9db5-2561028d5b2e/volumes"
Oct 30 22:33:50.241196 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.241137    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:33:50.265051 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265019    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-envoy\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265100    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8ea8884-75a7-4608-b491-40cb633aca64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea8884-75a7-4608-b491-40cb633aca64-istio-podinfo\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265161    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-credential-socket\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265238 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265231    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-data\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265462 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265288    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-istio-token\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265462 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265352    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-kube-api-access-hcj8h\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-kube-api-access-hcj8h\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265462 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265403    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-certs\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265511    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-socket\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265647 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265560    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ea8884-75a7-4608-b491-40cb633aca64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea8884-75a7-4608-b491-40cb633aca64-istiod-ca-cert\") pod \"c8ea8884-75a7-4608-b491-40cb633aca64\" (UID: \"c8ea8884-75a7-4608-b491-40cb633aca64\") "
Oct 30 22:33:50.265769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265714    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-credential-socket" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:50.265769 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.265731    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-data" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:50.266342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.266062    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-certs" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:50.266342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.266125    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ea8884-75a7-4608-b491-40cb633aca64-istiod-ca-cert" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:50.266342 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.266298    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-socket" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:50.266790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.266722    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.266790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.266749    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.266790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.266767    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.268158 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.268122    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-kube-api-access-hcj8h" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "kube-api-access-hcj8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:50.268387 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.268358    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8ea8884-75a7-4608-b491-40cb633aca64-istio-podinfo" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:50.269012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.268987    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-istio-token" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:50.269012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.268996    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-envoy" pod "c8ea8884-75a7-4608-b491-40cb633aca64" (UID: "c8ea8884-75a7-4608-b491-40cb633aca64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:50.353540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.353499    4282 generic.go:356] "Generic (PLEG): container finished" podID="c8ea8884-75a7-4608-b491-40cb633aca64" containerID="7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b" exitCode=0
Oct 30 22:33:50.353540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.353537    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"
Oct 30 22:33:50.353782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.353620    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" event={"ID":"c8ea8884-75a7-4608-b491-40cb633aca64","Type":"ContainerDied","Data":"7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b"}
Oct 30 22:33:50.353782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.353667    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9" event={"ID":"c8ea8884-75a7-4608-b491-40cb633aca64","Type":"ContainerDied","Data":"f470cd053c555ffdb4d0bfa49a5af1037197fe43ced495ac7756bf7c93a60a26"}
Oct 30 22:33:50.353782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.353698    4282 scope.go:122] "RemoveContainer" containerID="7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b"
Oct 30 22:33:50.369848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.369605    4282 scope.go:122] "RemoveContainer" containerID="7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b"
Oct 30 22:33:50.370892 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:33:50.370846    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b\": container with ID starting with 7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b not found: ID does not exist" containerID="7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b"
Oct 30 22:33:50.372823 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.370904    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b"} err="failed to get container status \"7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b\": rpc error: code = NotFound desc = could not find container \"7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b\": container with ID starting with 7f94b1276914fffda882cbc8503b888f1f585a8c58557a593930d4d2e37dfb9b not found: ID does not exist"
Oct 30 22:33:50.381665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.381530    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.381665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.381560    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcj8h\" (UniqueName: \"kubernetes.io/projected/c8ea8884-75a7-4608-b491-40cb633aca64-kube-api-access-hcj8h\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.381665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.381580    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.381665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.381604    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ea8884-75a7-4608-b491-40cb633aca64-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.381665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.381619    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ea8884-75a7-4608-b491-40cb633aca64-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.381665 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.381635    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ea8884-75a7-4608-b491-40cb633aca64-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:50.398072 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.398039    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"]
Oct 30 22:33:50.405672 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:50.403911    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f60026-openshift-default-6b5bc67c7f-r7rf9"]
Oct 30 22:33:51.220197 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:51.220166    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:33:51.492876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:51.492775    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" path="/var/lib/kubelet/pods/c8ea8884-75a7-4608-b491-40cb633aca64/volumes"
Oct 30 22:33:53.368806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:53.368764    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p" podUID="3900d25b-7d5e-4b35-826c-afaec7394e5b" containerName="payload-processing" containerID="cri-o://333bae14caaf449017644aa3b156abb342fb20d581eed251bde12d44445939be" gracePeriod=25
Oct 30 22:33:58.868934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:58.868897    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"]
Oct 30 22:33:58.869394 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:58.869182    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy" containerID="cri-o://ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff" gracePeriod=30
Oct 30 22:33:59.282514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.282460    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"]
Oct 30 22:33:59.282749 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.282722    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb" podUID="66680916-61a4-451c-b6d8-42a57d23705d" containerName="maas-api" containerID="cri-o://931a8f20f2e69b3254f858bdfe49c73e016dc15e4ab2b5ec779bd8d8e3ad420d" gracePeriod=30
Oct 30 22:33:59.419565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.419529    4282 generic.go:356] "Generic (PLEG): container finished" podID="66680916-61a4-451c-b6d8-42a57d23705d" containerID="931a8f20f2e69b3254f858bdfe49c73e016dc15e4ab2b5ec779bd8d8e3ad420d" exitCode=0
Oct 30 22:33:59.419746 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.419615    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb" event={"ID":"66680916-61a4-451c-b6d8-42a57d23705d","Type":"ContainerDied","Data":"931a8f20f2e69b3254f858bdfe49c73e016dc15e4ab2b5ec779bd8d8e3ad420d"}
Oct 30 22:33:59.595492 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.592603    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:33:59.754630 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.754596    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-maas-api-tls\") pod \"66680916-61a4-451c-b6d8-42a57d23705d\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") "
Oct 30 22:33:59.754810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.754650    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66680916-61a4-451c-b6d8-42a57d23705d-kube-api-access-k2vjx\" (UniqueName: \"kubernetes.io/projected/66680916-61a4-451c-b6d8-42a57d23705d-kube-api-access-k2vjx\") pod \"66680916-61a4-451c-b6d8-42a57d23705d\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") "
Oct 30 22:33:59.754810 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.754720    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-metrics-certs\") pod \"66680916-61a4-451c-b6d8-42a57d23705d\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") "
Oct 30 22:33:59.754973 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.754841    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66680916-61a4-451c-b6d8-42a57d23705d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66680916-61a4-451c-b6d8-42a57d23705d-openshift-service-ca\") pod \"66680916-61a4-451c-b6d8-42a57d23705d\" (UID: \"66680916-61a4-451c-b6d8-42a57d23705d\") "
Oct 30 22:33:59.755226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.755198    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66680916-61a4-451c-b6d8-42a57d23705d-openshift-service-ca" pod "66680916-61a4-451c-b6d8-42a57d23705d" (UID: "66680916-61a4-451c-b6d8-42a57d23705d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:59.757005 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.756981    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-maas-api-tls" pod "66680916-61a4-451c-b6d8-42a57d23705d" (UID: "66680916-61a4-451c-b6d8-42a57d23705d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:59.757102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.757027    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-metrics-certs" pod "66680916-61a4-451c-b6d8-42a57d23705d" (UID: "66680916-61a4-451c-b6d8-42a57d23705d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:33:59.757102 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.757054    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66680916-61a4-451c-b6d8-42a57d23705d-kube-api-access-k2vjx" pod "66680916-61a4-451c-b6d8-42a57d23705d" (UID: "66680916-61a4-451c-b6d8-42a57d23705d"). InnerVolumeSpecName "kube-api-access-k2vjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:59.856411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.856334    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:59.856411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.856360    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vjx\" (UniqueName: \"kubernetes.io/projected/66680916-61a4-451c-b6d8-42a57d23705d-kube-api-access-k2vjx\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:59.856411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.856372    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66680916-61a4-451c-b6d8-42a57d23705d-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:33:59.856411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.856382    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66680916-61a4-451c-b6d8-42a57d23705d-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.000037 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:33:59.999978    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main" containerID="cri-o://948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050" gracePeriod=45
Oct 30 22:34:00.021567 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.021524    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" containerID="cri-o://ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d" gracePeriod=45
Oct 30 22:34:00.135086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.135053    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": dial tcp 10.133.0.52:8000: connect: connection refused"
Oct 30 22:34:00.272975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.272946    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:34:00.284586 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.284554    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:34:00.429893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.429857    4282 generic.go:356] "Generic (PLEG): container finished" podID="a563a277-f347-4a67-b2a7-854282b5504a" containerID="ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d" exitCode=0
Oct 30 22:34:00.430075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.429967    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" event={"ID":"a563a277-f347-4a67-b2a7-854282b5504a","Type":"ContainerDied","Data":"ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d"}
Oct 30 22:34:00.430075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.430018    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q" event={"ID":"a563a277-f347-4a67-b2a7-854282b5504a","Type":"ContainerDied","Data":"6ae6c2b3d530244e9652b1e3726c850102f146caf48ae1938a70284a55686af1"}
Oct 30 22:34:00.430075 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.430041    4282 scope.go:122] "RemoveContainer" containerID="ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d"
Oct 30 22:34:00.430277 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.430154    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"
Oct 30 22:34:00.432111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.432077    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"
Oct 30 22:34:00.432111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.432089    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb" event={"ID":"66680916-61a4-451c-b6d8-42a57d23705d","Type":"ContainerDied","Data":"d46270278dbea86eb5e36e92381c1065b47baa18e13c62a1a40e626c4765e8d1"}
Oct 30 22:34:00.433928 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.433900    4282 generic.go:356] "Generic (PLEG): container finished" podID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerID="948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050" exitCode=0
Oct 30 22:34:00.434044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.433944    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" event={"ID":"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75","Type":"ContainerDied","Data":"948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050"}
Oct 30 22:34:00.434044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.433971    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427" event={"ID":"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75","Type":"ContainerDied","Data":"9308e94ffa6c9028da0a1afeab2efa8b06a2f02a56cd29b3461be05f3fb3b6fd"}
Oct 30 22:34:00.434044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.434028    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"
Oct 30 22:34:00.441637 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.441615    4282 scope.go:122] "RemoveContainer" containerID="ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d"
Oct 30 22:34:00.442028 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:00.442006    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d\": container with ID starting with ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d not found: ID does not exist" containerID="ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d"
Oct 30 22:34:00.442092 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.442035    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d"} err="failed to get container status \"ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d\": rpc error: code = NotFound desc = could not find container \"ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d\": container with ID starting with ee8ef2b0126cadf0f8d2248f869256bc865a63699115ad9c8db0af301fec1b8d not found: ID does not exist"
Oct 30 22:34:00.442092 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.442052    4282 scope.go:122] "RemoveContainer" containerID="931a8f20f2e69b3254f858bdfe49c73e016dc15e4ab2b5ec779bd8d8e3ad420d"
Oct 30 22:34:00.453313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.453281    4282 scope.go:122] "RemoveContainer" containerID="948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050"
Oct 30 22:34:00.462599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.462576    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"]
Oct 30 22:34:00.463911 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.463886    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-home\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-home\") pod \"a563a277-f347-4a67-b2a7-854282b5504a\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") "
Oct 30 22:34:00.464047 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.463968    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-model-cache\") pod \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") "
Oct 30 22:34:00.464127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464064    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-kube-api-access-fkd4p\" (UniqueName: \"kubernetes.io/projected/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-kube-api-access-fkd4p\") pod \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") "
Oct 30 22:34:00.464127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464121    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-dshm\") pod \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") "
Oct 30 22:34:00.464269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464171    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tls-certs\") pod \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") "
Oct 30 22:34:00.464269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464207    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-dshm\") pod \"a563a277-f347-4a67-b2a7-854282b5504a\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") "
Oct 30 22:34:00.464269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464238    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-tmp-dir\") pod \"a563a277-f347-4a67-b2a7-854282b5504a\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") "
Oct 30 22:34:00.464269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464265    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a563a277-f347-4a67-b2a7-854282b5504a-kube-api-access-xqvp9\" (UniqueName: \"kubernetes.io/projected/a563a277-f347-4a67-b2a7-854282b5504a-kube-api-access-xqvp9\") pod \"a563a277-f347-4a67-b2a7-854282b5504a\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") "
Oct 30 22:34:00.464563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464311    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-model-cache\") pod \"a563a277-f347-4a67-b2a7-854282b5504a\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") "
Oct 30 22:34:00.464563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464425    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-home\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-home\") pod \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") "
Oct 30 22:34:00.464563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464505    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a563a277-f347-4a67-b2a7-854282b5504a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a563a277-f347-4a67-b2a7-854282b5504a-tls-certs\") pod \"a563a277-f347-4a67-b2a7-854282b5504a\" (UID: \"a563a277-f347-4a67-b2a7-854282b5504a\") "
Oct 30 22:34:00.464563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.464533    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tmp-dir\") pod \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\" (UID: \"ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75\") "
Oct 30 22:34:00.465214 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.465189    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tmp-dir" pod "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" (UID: "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:00.465948 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.465924    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-model-cache" pod "a563a277-f347-4a67-b2a7-854282b5504a" (UID: "a563a277-f347-4a67-b2a7-854282b5504a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:00.466173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.466148    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-tmp-dir" pod "a563a277-f347-4a67-b2a7-854282b5504a" (UID: "a563a277-f347-4a67-b2a7-854282b5504a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:00.466547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.466522    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-home" pod "a563a277-f347-4a67-b2a7-854282b5504a" (UID: "a563a277-f347-4a67-b2a7-854282b5504a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:00.467718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.467686    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tls-certs" pod "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" (UID: "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:00.467718 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.467707    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a563a277-f347-4a67-b2a7-854282b5504a-kube-api-access-xqvp9" pod "a563a277-f347-4a67-b2a7-854282b5504a" (UID: "a563a277-f347-4a67-b2a7-854282b5504a"). InnerVolumeSpecName "kube-api-access-xqvp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:00.467869 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.467717    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-06ac2e-7cf69d8445-rpvmb"]
Oct 30 22:34:00.467869 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.467782    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-dshm" pod "a563a277-f347-4a67-b2a7-854282b5504a" (UID: "a563a277-f347-4a67-b2a7-854282b5504a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:00.468214 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.468193    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a563a277-f347-4a67-b2a7-854282b5504a-tls-certs" pod "a563a277-f347-4a67-b2a7-854282b5504a" (UID: "a563a277-f347-4a67-b2a7-854282b5504a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:00.468500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.468445    4282 scope.go:122] "RemoveContainer" containerID="948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050"
Oct 30 22:34:00.468621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.468601    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-model-cache" pod "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" (UID: "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:00.468833 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.468800    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-home" pod "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" (UID: "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:00.469147 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:00.469127    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050\": container with ID starting with 948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050 not found: ID does not exist" containerID="948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050"
Oct 30 22:34:00.469225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.469157    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050"} err="failed to get container status \"948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050\": rpc error: code = NotFound desc = could not find container \"948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050\": container with ID starting with 948e03328a720caf64fd1edd389fa51d9e88be574ab4600e8478e17fd1082050 not found: ID does not exist"
Oct 30 22:34:00.470296 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.470275    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-kube-api-access-fkd4p" pod "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" (UID: "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75"). InnerVolumeSpecName "kube-api-access-fkd4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:00.470414 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.470396    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-dshm" pod "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" (UID: "ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:00.516846 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.516805    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p" podUID="3900d25b-7d5e-4b35-826c-afaec7394e5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:00.566447 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566411    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566447 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566446    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566459    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566506    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566522    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqvp9\" (UniqueName: \"kubernetes.io/projected/a563a277-f347-4a67-b2a7-854282b5504a-kube-api-access-xqvp9\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566538    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566552    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566565    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a563a277-f347-4a67-b2a7-854282b5504a-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566578    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566592    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a563a277-f347-4a67-b2a7-854282b5504a-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566605    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.566714 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.566617    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkd4p\" (UniqueName: \"kubernetes.io/projected/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75-kube-api-access-fkd4p\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:00.773743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.773212    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"]
Oct 30 22:34:00.778768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.778733    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/premium-w0-1529dc-kserve-76b65d7c68-wcc2q"]
Oct 30 22:34:00.785263 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.785223    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"]
Oct 30 22:34:00.788269 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:00.788237    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-w0-1529dc-kserve-84778c9587-rt427"]
Oct 30 22:34:01.463486 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:01.463442    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66680916-61a4-451c-b6d8-42a57d23705d" path="/var/lib/kubelet/pods/66680916-61a4-451c-b6d8-42a57d23705d/volumes"
Oct 30 22:34:01.463918 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:01.463904    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a563a277-f347-4a67-b2a7-854282b5504a" path="/var/lib/kubelet/pods/a563a277-f347-4a67-b2a7-854282b5504a/volumes"
Oct 30 22:34:01.464287 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:01.464275    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" path="/var/lib/kubelet/pods/ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75/volumes"
Oct 30 22:34:04.184399 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.184285    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230337    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-certs\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230425    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-kube-api-access-97wj4\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-kube-api-access-97wj4\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230516    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-credential-socket\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230552    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-socket\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230611    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54455775-f1d8-4e14-afe1-fd8ad99fb188-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54455775-f1d8-4e14-afe1-fd8ad99fb188-istiod-ca-cert\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230714    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-envoy\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230847    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-token\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-token\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230876    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-data\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.231525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.230907    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-podinfo\") pod \"54455775-f1d8-4e14-afe1-fd8ad99fb188\" (UID: \"54455775-f1d8-4e14-afe1-fd8ad99fb188\") "
Oct 30 22:34:04.232175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.232093    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-socket" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:04.232400 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.232344    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-certs" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:04.241188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.236890    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-podinfo" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:04.241188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.237210    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54455775-f1d8-4e14-afe1-fd8ad99fb188-istiod-ca-cert" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:04.241188 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.240196    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-envoy" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:04.241512 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.241384    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-credential-socket" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:04.241942 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.241632    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-data" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:04.244789 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.244755    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-kube-api-access-97wj4" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "kube-api-access-97wj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:04.247540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.247514    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-token" pod "54455775-f1d8-4e14-afe1-fd8ad99fb188" (UID: "54455775-f1d8-4e14-afe1-fd8ad99fb188"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:04.332524 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332451    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.332524 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332519    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.332848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332534    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.332848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332546    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54455775-f1d8-4e14-afe1-fd8ad99fb188-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.332848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332558    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.332848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332570    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97wj4\" (UniqueName: \"kubernetes.io/projected/54455775-f1d8-4e14-afe1-fd8ad99fb188-kube-api-access-97wj4\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.332848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332583    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.332848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332596    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54455775-f1d8-4e14-afe1-fd8ad99fb188-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.332848 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.332609    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54455775-f1d8-4e14-afe1-fd8ad99fb188-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:04.461544 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.461514    4282 generic.go:356] "Generic (PLEG): container finished" podID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerID="ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff" exitCode=0
Oct 30 22:34:04.461694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.461560    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"
Oct 30 22:34:04.461694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.461600    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" event={"ID":"54455775-f1d8-4e14-afe1-fd8ad99fb188","Type":"ContainerDied","Data":"ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff"}
Oct 30 22:34:04.461694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.461643    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl" event={"ID":"54455775-f1d8-4e14-afe1-fd8ad99fb188","Type":"ContainerDied","Data":"6b6aa100e426d12b8a023a338ef047801033c61dc01a3b226ad208e982f6cbef"}
Oct 30 22:34:04.461694 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.461668    4282 scope.go:122] "RemoveContainer" containerID="ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff"
Oct 30 22:34:04.474721 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.474704    4282 scope.go:122] "RemoveContainer" containerID="ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff"
Oct 30 22:34:04.478899 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:04.478870    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff\": container with ID starting with ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff not found: ID does not exist" containerID="ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff"
Oct 30 22:34:04.479011 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.478911    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff"} err="failed to get container status \"ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff\": rpc error: code = NotFound desc = could not find container \"ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff\": container with ID starting with ee257d74f5e243519c1867e3d1b548c478f056b2fe6e815e6b81ed26801db9ff not found: ID does not exist"
Oct 30 22:34:04.489849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.489823    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"]
Oct 30 22:34:04.495579 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.495558    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gx9nl"]
Oct 30 22:34:04.650820 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:04.650781    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:34:04.912607 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:04.912526    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"]
Oct 30 22:34:05.463673 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:05.463638    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" path="/var/lib/kubelet/pods/54455775-f1d8-4e14-afe1-fd8ad99fb188/volumes"
Oct 30 22:34:10.516523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:10.516492    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p" podUID="3900d25b-7d5e-4b35-826c-afaec7394e5b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:10.835836 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:10.835723    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"]
Oct 30 22:34:10.836012 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:10.835959    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy" containerID="cri-o://7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c" gracePeriod=30
Oct 30 22:34:10.840098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:10.839592    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"]
Oct 30 22:34:10.840098 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:10.839789    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy" containerID="cri-o://d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4" gracePeriod=30
Oct 30 22:34:12.651330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:12.651295    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:34:13.510834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.510797    4282 generic.go:356] "Generic (PLEG): container finished" podID="3900d25b-7d5e-4b35-826c-afaec7394e5b" containerID="333bae14caaf449017644aa3b156abb342fb20d581eed251bde12d44445939be" exitCode=0
Oct 30 22:34:13.511048 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.510872    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p" event={"ID":"3900d25b-7d5e-4b35-826c-afaec7394e5b","Type":"ContainerDied","Data":"333bae14caaf449017644aa3b156abb342fb20d581eed251bde12d44445939be"}
Oct 30 22:34:13.632083 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.632019    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:34:13.748385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.748355    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3900d25b-7d5e-4b35-826c-afaec7394e5b-kube-api-access-mj7bz\" (UniqueName: \"kubernetes.io/projected/3900d25b-7d5e-4b35-826c-afaec7394e5b-kube-api-access-mj7bz\") pod \"3900d25b-7d5e-4b35-826c-afaec7394e5b\" (UID: \"3900d25b-7d5e-4b35-826c-afaec7394e5b\") "
Oct 30 22:34:13.748940 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.748452    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3900d25b-7d5e-4b35-826c-afaec7394e5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3900d25b-7d5e-4b35-826c-afaec7394e5b-plugins-config-volume\") pod \"3900d25b-7d5e-4b35-826c-afaec7394e5b\" (UID: \"3900d25b-7d5e-4b35-826c-afaec7394e5b\") "
Oct 30 22:34:13.749079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.749052    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3900d25b-7d5e-4b35-826c-afaec7394e5b-plugins-config-volume" pod "3900d25b-7d5e-4b35-826c-afaec7394e5b" (UID: "3900d25b-7d5e-4b35-826c-afaec7394e5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:13.751358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.751330    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3900d25b-7d5e-4b35-826c-afaec7394e5b-kube-api-access-mj7bz" pod "3900d25b-7d5e-4b35-826c-afaec7394e5b" (UID: "3900d25b-7d5e-4b35-826c-afaec7394e5b"). InnerVolumeSpecName "kube-api-access-mj7bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:13.849482 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.849447    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj7bz\" (UniqueName: \"kubernetes.io/projected/3900d25b-7d5e-4b35-826c-afaec7394e5b-kube-api-access-mj7bz\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:13.849628 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:13.849499    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3900d25b-7d5e-4b35-826c-afaec7394e5b-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:14.525516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:14.525344    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p" event={"ID":"3900d25b-7d5e-4b35-826c-afaec7394e5b","Type":"ContainerDied","Data":"4ea64eb827fa91f68d1f6fb28cdc6769a187f70453ea3723c97cefb056d12f3c"}
Oct 30 22:34:14.525876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:14.525847    4282 scope.go:122] "RemoveContainer" containerID="333bae14caaf449017644aa3b156abb342fb20d581eed251bde12d44445939be"
Oct 30 22:34:14.527797 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:14.526624    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"
Oct 30 22:34:14.566522 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:14.566497    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"]
Oct 30 22:34:14.577872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:14.577849    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0f73c9-669647cdbf-42t7p"]
Oct 30 22:34:15.463278 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:15.463235    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3900d25b-7d5e-4b35-826c-afaec7394e5b" path="/var/lib/kubelet/pods/3900d25b-7d5e-4b35-826c-afaec7394e5b/volumes"
Oct 30 22:34:16.111210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.111137    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 22:34:16.129321 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.129291    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 22:34:16.176359 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176327    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-socket\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.176555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176413    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-token\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-token\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.176555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176497    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-data\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.176555 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176530    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e91cee75-83ba-468a-a4c0-9c40820adedc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e91cee75-83ba-468a-a4c0-9c40820adedc-istiod-ca-cert\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.176751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176576    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-podinfo\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.176751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176610    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-kube-api-access-fdtk2\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-kube-api-access-fdtk2\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.176751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176639    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-token\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-token\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.176751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176644    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-socket" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.176751 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176674    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-kube-api-access-hnbpz\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-kube-api-access-hnbpz\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.177119 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.176824    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-data" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.177239 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177213    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-credential-socket\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.177379 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177297    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91cee75-83ba-468a-a4c0-9c40820adedc-istiod-ca-cert" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:16.177514 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177461    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-credential-socket" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.177763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177620    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-certs\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.177763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177662    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-certs\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.177763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177706    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-data\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.177763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177734    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b416f91-5c95-43cd-bdc6-402bca298e02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b416f91-5c95-43cd-bdc6-402bca298e02-istiod-ca-cert\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.178003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177775    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-envoy\") pod \"4b416f91-5c95-43cd-bdc6-402bca298e02\" (UID: \"4b416f91-5c95-43cd-bdc6-402bca298e02\") "
Oct 30 22:34:16.178003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177812    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-certs" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.178003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177820    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-podinfo\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.178003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177877    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-envoy\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.178003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177913    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-credential-socket\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.178003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.177943    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-socket\") pod \"e91cee75-83ba-468a-a4c0-9c40820adedc\" (UID: \"e91cee75-83ba-468a-a4c0-9c40820adedc\") "
Oct 30 22:34:16.178519 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.178495    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.178610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.178526    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.178610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.178542    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.178610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.178557    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.178610 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.178570    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e91cee75-83ba-468a-a4c0-9c40820adedc-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.178849 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.178786    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-socket" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.179309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.179278    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-credential-socket" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.179575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.179494    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-data" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.179951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.179913    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b416f91-5c95-43cd-bdc6-402bca298e02-istiod-ca-cert" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:16.180144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.180113    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-certs" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.180144 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.180116    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-kube-api-access-hnbpz" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "kube-api-access-hnbpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:16.180768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.180744    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-token" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:16.180922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.180861    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-token" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:16.180922 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.180908    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-kube-api-access-fdtk2" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "kube-api-access-fdtk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:16.181599 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.181574    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-envoy" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.181782 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.181764    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-podinfo" pod "e91cee75-83ba-468a-a4c0-9c40820adedc" (UID: "e91cee75-83ba-468a-a4c0-9c40820adedc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:16.182101 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.182078    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-envoy" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:16.182800 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.182779    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-podinfo" pod "4b416f91-5c95-43cd-bdc6-402bca298e02" (UID: "4b416f91-5c95-43cd-bdc6-402bca298e02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:16.279300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279253    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279296    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279312    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279326    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279339    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279351    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279364    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdtk2\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-kube-api-access-fdtk2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279376    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e91cee75-83ba-468a-a4c0-9c40820adedc-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279385    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnbpz\" (UniqueName: \"kubernetes.io/projected/4b416f91-5c95-43cd-bdc6-402bca298e02-kube-api-access-hnbpz\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279395    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e91cee75-83ba-468a-a4c0-9c40820adedc-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279403    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279411    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b416f91-5c95-43cd-bdc6-402bca298e02-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.279619 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.279419    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b416f91-5c95-43cd-bdc6-402bca298e02-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:16.540257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.540220    4282 generic.go:356] "Generic (PLEG): container finished" podID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerID="7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c" exitCode=0
Oct 30 22:34:16.540765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.540274    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"
Oct 30 22:34:16.540765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.540321    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" event={"ID":"4b416f91-5c95-43cd-bdc6-402bca298e02","Type":"ContainerDied","Data":"7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c"}
Oct 30 22:34:16.540765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.540382    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff" event={"ID":"4b416f91-5c95-43cd-bdc6-402bca298e02","Type":"ContainerDied","Data":"566cba97de91bf9be307b63a951c5b1a1945485def1ffe60b3dec7e3728671cf"}
Oct 30 22:34:16.540765 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.540416    4282 scope.go:122] "RemoveContainer" containerID="7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c"
Oct 30 22:34:16.541890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.541766    4282 generic.go:356] "Generic (PLEG): container finished" podID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerID="d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4" exitCode=0
Oct 30 22:34:16.541890 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.541814    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"
Oct 30 22:34:16.542082 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.541900    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" event={"ID":"e91cee75-83ba-468a-a4c0-9c40820adedc","Type":"ContainerDied","Data":"d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4"}
Oct 30 22:34:16.542082 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.542004    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4" event={"ID":"e91cee75-83ba-468a-a4c0-9c40820adedc","Type":"ContainerDied","Data":"a711ceb4365b6308406098cbd161cd6c4c3c262fa02ac6ae560610214d44b997"}
Oct 30 22:34:16.557328 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.557280    4282 scope.go:122] "RemoveContainer" containerID="34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46"
Oct 30 22:34:16.566901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.566881    4282 scope.go:122] "RemoveContainer" containerID="7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c"
Oct 30 22:34:16.567159 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:16.567140    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c\": container with ID starting with 7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c not found: ID does not exist" containerID="7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c"
Oct 30 22:34:16.567210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.567169    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c"} err="failed to get container status \"7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c\": rpc error: code = NotFound desc = could not find container \"7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c\": container with ID starting with 7497a194a2263bac8d46cf250d5430b9a5597b6916d5c103d539578af046fe4c not found: ID does not exist"
Oct 30 22:34:16.567210 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.567185    4282 scope.go:122] "RemoveContainer" containerID="34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46"
Oct 30 22:34:16.567442 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:16.567423    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46\": container with ID starting with 34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46 not found: ID does not exist" containerID="34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46"
Oct 30 22:34:16.567534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.567452    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46"} err="failed to get container status \"34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46\": rpc error: code = NotFound desc = could not find container \"34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46\": container with ID starting with 34c6ae43b9b5ee84aee793a4ccaa22e117fdce9cd0c0384c2053e3e41bd1ee46 not found: ID does not exist"
Oct 30 22:34:16.567534 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.567489    4282 scope.go:122] "RemoveContainer" containerID="d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4"
Oct 30 22:34:16.574908 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.574888    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"]
Oct 30 22:34:16.576734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.576715    4282 scope.go:122] "RemoveContainer" containerID="d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4"
Oct 30 22:34:16.577011 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:16.576991    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4\": container with ID starting with d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4 not found: ID does not exist" containerID="d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4"
Oct 30 22:34:16.577090 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.577021    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4"} err="failed to get container status \"d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4\": rpc error: code = NotFound desc = could not find container \"d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4\": container with ID starting with d21dd8415ab997ed14f107832782680596342a933f0d046f54f27def469637a4 not found: ID does not exist"
Oct 30 22:34:16.583039 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.583016    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2lvs4"]
Oct 30 22:34:16.595552 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.595530    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"]
Oct 30 22:34:16.603578 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:16.603557    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-778ff"]
Oct 30 22:34:17.463298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:17.463264    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" path="/var/lib/kubelet/pods/4b416f91-5c95-43cd-bdc6-402bca298e02/volumes"
Oct 30 22:34:17.463802 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:17.463786    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" path="/var/lib/kubelet/pods/e91cee75-83ba-468a-a4c0-9c40820adedc/volumes"
Oct 30 22:34:18.254111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:18.254074    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"]
Oct 30 22:34:18.271134 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:18.271105    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"]
Oct 30 22:34:19.933841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:19.933803    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main" containerID="cri-o://38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6" gracePeriod=45
Oct 30 22:34:20.204280 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.204256    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:34:20.338701 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.338670    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-dshm\") pod \"deecd927-414a-41ec-9341-89cc0af48c7f\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") "
Oct 30 22:34:20.338898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.338723    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deecd927-414a-41ec-9341-89cc0af48c7f-kube-api-access-tscn2\" (UniqueName: \"kubernetes.io/projected/deecd927-414a-41ec-9341-89cc0af48c7f-kube-api-access-tscn2\") pod \"deecd927-414a-41ec-9341-89cc0af48c7f\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") "
Oct 30 22:34:20.338898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.338780    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-model-cache\") pod \"deecd927-414a-41ec-9341-89cc0af48c7f\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") "
Oct 30 22:34:20.338898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.338838    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-home\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-home\") pod \"deecd927-414a-41ec-9341-89cc0af48c7f\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") "
Oct 30 22:34:20.338898 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.338885    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-tmp-dir\") pod \"deecd927-414a-41ec-9341-89cc0af48c7f\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") "
Oct 30 22:34:20.339149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.338961    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deecd927-414a-41ec-9341-89cc0af48c7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/deecd927-414a-41ec-9341-89cc0af48c7f-tls-certs\") pod \"deecd927-414a-41ec-9341-89cc0af48c7f\" (UID: \"deecd927-414a-41ec-9341-89cc0af48c7f\") "
Oct 30 22:34:20.339423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.339227    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-home" pod "deecd927-414a-41ec-9341-89cc0af48c7f" (UID: "deecd927-414a-41ec-9341-89cc0af48c7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.339423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.339324    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-tmp-dir" pod "deecd927-414a-41ec-9341-89cc0af48c7f" (UID: "deecd927-414a-41ec-9341-89cc0af48c7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.339423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.339386    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-model-cache" pod "deecd927-414a-41ec-9341-89cc0af48c7f" (UID: "deecd927-414a-41ec-9341-89cc0af48c7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.339702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.339582    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.339702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.339605    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.339702 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.339621    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.341865 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.341832    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-dshm" pod "deecd927-414a-41ec-9341-89cc0af48c7f" (UID: "deecd927-414a-41ec-9341-89cc0af48c7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:20.341989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.341922    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deecd927-414a-41ec-9341-89cc0af48c7f-tls-certs" pod "deecd927-414a-41ec-9341-89cc0af48c7f" (UID: "deecd927-414a-41ec-9341-89cc0af48c7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:20.341989 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.341961    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deecd927-414a-41ec-9341-89cc0af48c7f-kube-api-access-tscn2" pod "deecd927-414a-41ec-9341-89cc0af48c7f" (UID: "deecd927-414a-41ec-9341-89cc0af48c7f"). InnerVolumeSpecName "kube-api-access-tscn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:20.440617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.440543    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deecd927-414a-41ec-9341-89cc0af48c7f-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.440617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.440570    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tscn2\" (UniqueName: \"kubernetes.io/projected/deecd927-414a-41ec-9341-89cc0af48c7f-kube-api-access-tscn2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.440617 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.440582    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deecd927-414a-41ec-9341-89cc0af48c7f-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:20.575500 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.575449    4282 generic.go:356] "Generic (PLEG): container finished" podID="deecd927-414a-41ec-9341-89cc0af48c7f" containerID="38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6" exitCode=0
Oct 30 22:34:20.575658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.575515    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"
Oct 30 22:34:20.575658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.575535    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" event={"ID":"deecd927-414a-41ec-9341-89cc0af48c7f","Type":"ContainerDied","Data":"38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6"}
Oct 30 22:34:20.575658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.575577    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb" event={"ID":"deecd927-414a-41ec-9341-89cc0af48c7f","Type":"ContainerDied","Data":"626e97090ee253247a427f9275922b416f5ef1d35712f1216b66f67de16e551e"}
Oct 30 22:34:20.575658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.575595    4282 scope.go:122] "RemoveContainer" containerID="38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6"
Oct 30 22:34:20.585225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.585206    4282 scope.go:122] "RemoveContainer" containerID="38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6"
Oct 30 22:34:20.585527 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:20.585507    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6\": container with ID starting with 38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6 not found: ID does not exist" containerID="38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6"
Oct 30 22:34:20.585596 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.585537    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6"} err="failed to get container status \"38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6\": rpc error: code = NotFound desc = could not find container \"38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6\": container with ID starting with 38e12c9a4d1bb1de568e6bcf71f0debb28e83c189279478ab6e56487815669e6 not found: ID does not exist"
Oct 30 22:34:20.597132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.597109    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"]
Oct 30 22:34:20.599798 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:20.599776    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-w3-ceb36a-kserve-56fc566456-8k4mb"]
Oct 30 22:34:21.464631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:21.464601    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" path="/var/lib/kubelet/pods/deecd927-414a-41ec-9341-89cc0af48c7f/volumes"
Oct 30 22:34:23.255247 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:23.255157    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz" podUID="1aa65e87-cda6-468f-b669-44b5121ed498" containerName="payload-processing" containerID="cri-o://40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0" gracePeriod=25
Oct 30 22:34:23.271725 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:23.271677    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96" podUID="adb62d9b-f7b2-4118-8669-cda1e51557e6" containerName="payload-pre-processing" containerID="cri-o://1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7" gracePeriod=25
Oct 30 22:34:28.236391 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:28.236357    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz" podUID="1aa65e87-cda6-468f-b669-44b5121ed498" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:28.236897 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:28.236357    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96" podUID="adb62d9b-f7b2-4118-8669-cda1e51557e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:30.653735 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:30.653705    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:34:30.873894 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:30.873851    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"]
Oct 30 22:34:30.874132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:30.874106    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy" containerID="cri-o://ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a" gracePeriod=30
Oct 30 22:34:34.648758 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:34.648721    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:34:35.183859 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:35.183829    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"]
Oct 30 22:34:35.184056 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:35.184027    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy" containerID="cri-o://699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425" gracePeriod=30
Oct 30 22:34:36.160057 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.160032    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:34:36.226788 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.226754    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-envoy\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.226956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.226819    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-socket\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.226956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.226842    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-kube-api-access-jnvdj\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-kube-api-access-jnvdj\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.226956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.226867    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-token\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-token\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.226956 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.226899    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istiod-ca-cert\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.227208 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.226993    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-credential-socket\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.227208 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227047    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-podinfo\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.227208 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227100    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-data\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.227382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227169    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-socket" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:36.227382 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227212    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-certs\") pod \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\" (UID: \"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d\") "
Oct 30 22:34:36.227525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227422    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istiod-ca-cert" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:36.227525 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227436    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-certs" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:36.227737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227650    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-data" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:36.227737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227719    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-credential-socket" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:36.227944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227918    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.228009 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227942    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.228009 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227958    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.228009 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.227972    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.229155 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.229129    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-envoy" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:36.229389 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.229367    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-podinfo" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:36.229445 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.229417    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-kube-api-access-jnvdj" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "kube-api-access-jnvdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:36.229547 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.229526    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-token" pod "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" (UID: "ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:36.329349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.329311    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.329349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.329348    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnvdj\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-kube-api-access-jnvdj\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.329625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.329363    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.329625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.329373    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.329625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.329381    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:36.669634 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.669599    4282 generic.go:356] "Generic (PLEG): container finished" podID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerID="ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a" exitCode=0
Oct 30 22:34:36.669994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.669641    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" event={"ID":"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d","Type":"ContainerDied","Data":"ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a"}
Oct 30 22:34:36.669994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.669663    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"
Oct 30 22:34:36.669994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.669701    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq" event={"ID":"ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d","Type":"ContainerDied","Data":"33ace5388edd6545fb4e7b3166edd3cdfea00711e5669da897a62aced046bb90"}
Oct 30 22:34:36.669994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.669730    4282 scope.go:122] "RemoveContainer" containerID="ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a"
Oct 30 22:34:36.680300 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.680280    4282 scope.go:122] "RemoveContainer" containerID="ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a"
Oct 30 22:34:36.680623 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:36.680606    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a\": container with ID starting with ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a not found: ID does not exist" containerID="ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a"
Oct 30 22:34:36.680680 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.680631    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a"} err="failed to get container status \"ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a\": rpc error: code = NotFound desc = could not find container \"ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a\": container with ID starting with ec93528df819bc6e2750b96d2cebfd4d508b9b3e01c176bcff01ff02ef51d85a not found: ID does not exist"
Oct 30 22:34:36.699575 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.699545    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"]
Oct 30 22:34:36.703019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:36.702998    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-qwtgq"]
Oct 30 22:34:37.464055 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:37.464021    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" path="/var/lib/kubelet/pods/ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d/volumes"
Oct 30 22:34:38.236987 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:38.236955    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz" podUID="1aa65e87-cda6-468f-b669-44b5121ed498" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:38.237181 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:38.236960    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96" podUID="adb62d9b-f7b2-4118-8669-cda1e51557e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:40.437204 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.437182    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 22:34:40.474150 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474095    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-credential-socket\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474150 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474132    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-kube-api-access-fm9sh\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-kube-api-access-fm9sh\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474236    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-data\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474262    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-socket\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474355    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-podinfo\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474359    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-credential-socket" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:40.474497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474384    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-certs\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474497 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474444    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istiod-ca-cert\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474511    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-envoy\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474548    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-token\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-token\") pod \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\" (UID: \"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3\") "
Oct 30 22:34:40.474734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474598    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-data" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:40.474734 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474612    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-socket" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:40.475113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474752    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-certs" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:40.475113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.474892    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istiod-ca-cert" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:40.475113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.475038    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.475113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.475059    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.475113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.475075    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.475113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.475089    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.475113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.475102    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.477302 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.477273    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-envoy" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:40.477423 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.477371    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-token" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:40.477733 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.477697    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-kube-api-access-fm9sh" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "kube-api-access-fm9sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:40.477958 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.477939    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-podinfo" pod "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" (UID: "b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:40.575906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.575872    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.575906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.575900    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.575906 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.575909    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.576118 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.575918    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm9sh\" (UniqueName: \"kubernetes.io/projected/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3-kube-api-access-fm9sh\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:40.696112 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.696078    4282 generic.go:356] "Generic (PLEG): container finished" podID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerID="699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425" exitCode=0
Oct 30 22:34:40.696298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.696126    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"
Oct 30 22:34:40.696298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.696154    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" event={"ID":"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3","Type":"ContainerDied","Data":"699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425"}
Oct 30 22:34:40.696298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.696211    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85" event={"ID":"b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3","Type":"ContainerDied","Data":"c94f071f097e22a33a0ef347701d9ff271f9bef31f708e7d90dacdeb3c4af432"}
Oct 30 22:34:40.696298 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.696235    4282 scope.go:122] "RemoveContainer" containerID="699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425"
Oct 30 22:34:40.706879 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.706861    4282 scope.go:122] "RemoveContainer" containerID="699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425"
Oct 30 22:34:40.707141 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:40.707120    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425\": container with ID starting with 699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425 not found: ID does not exist" containerID="699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425"
Oct 30 22:34:40.707208 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.707155    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425"} err="failed to get container status \"699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425\": rpc error: code = NotFound desc = could not find container \"699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425\": container with ID starting with 699ed4818c2043a72f3a4946732a9b692816470fd4a58a2702aef2c1dc3af425 not found: ID does not exist"
Oct 30 22:34:40.718509 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.718483    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"]
Oct 30 22:34:40.727838 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:40.727789    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-lfh85"]
Oct 30 22:34:41.463346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:41.463312    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" path="/var/lib/kubelet/pods/b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3/volumes"
Oct 30 22:34:41.529299 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:41.529266    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"]
Oct 30 22:34:43.527931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.527872    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:34:43.570131 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.570106    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:34:43.604143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.604119    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb62d9b-f7b2-4118-8669-cda1e51557e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb62d9b-f7b2-4118-8669-cda1e51557e6-plugins-config-volume\") pod \"adb62d9b-f7b2-4118-8669-cda1e51557e6\" (UID: \"adb62d9b-f7b2-4118-8669-cda1e51557e6\") "
Oct 30 22:34:43.604309 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.604169    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa65e87-cda6-468f-b669-44b5121ed498-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa65e87-cda6-468f-b669-44b5121ed498-plugins-config-volume\") pod \"1aa65e87-cda6-468f-b669-44b5121ed498\" (UID: \"1aa65e87-cda6-468f-b669-44b5121ed498\") "
Oct 30 22:34:43.604395 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.604298    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb62d9b-f7b2-4118-8669-cda1e51557e6-kube-api-access-wkqdq\" (UniqueName: \"kubernetes.io/projected/adb62d9b-f7b2-4118-8669-cda1e51557e6-kube-api-access-wkqdq\") pod \"adb62d9b-f7b2-4118-8669-cda1e51557e6\" (UID: \"adb62d9b-f7b2-4118-8669-cda1e51557e6\") "
Oct 30 22:34:43.604395 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.604374    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa65e87-cda6-468f-b669-44b5121ed498-kube-api-access-tj8lq\" (UniqueName: \"kubernetes.io/projected/1aa65e87-cda6-468f-b669-44b5121ed498-kube-api-access-tj8lq\") pod \"1aa65e87-cda6-468f-b669-44b5121ed498\" (UID: \"1aa65e87-cda6-468f-b669-44b5121ed498\") "
Oct 30 22:34:43.604560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.604539    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb62d9b-f7b2-4118-8669-cda1e51557e6-plugins-config-volume" pod "adb62d9b-f7b2-4118-8669-cda1e51557e6" (UID: "adb62d9b-f7b2-4118-8669-cda1e51557e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:43.604621 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.604576    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa65e87-cda6-468f-b669-44b5121ed498-plugins-config-volume" pod "1aa65e87-cda6-468f-b669-44b5121ed498" (UID: "1aa65e87-cda6-468f-b669-44b5121ed498"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:43.604822 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.604805    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb62d9b-f7b2-4118-8669-cda1e51557e6-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:43.604905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.604829    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa65e87-cda6-468f-b669-44b5121ed498-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:43.606743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.606712    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa65e87-cda6-468f-b669-44b5121ed498-kube-api-access-tj8lq" pod "1aa65e87-cda6-468f-b669-44b5121ed498" (UID: "1aa65e87-cda6-468f-b669-44b5121ed498"). InnerVolumeSpecName "kube-api-access-tj8lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:43.606743 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.606715    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb62d9b-f7b2-4118-8669-cda1e51557e6-kube-api-access-wkqdq" pod "adb62d9b-f7b2-4118-8669-cda1e51557e6" (UID: "adb62d9b-f7b2-4118-8669-cda1e51557e6"). InnerVolumeSpecName "kube-api-access-wkqdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:43.705892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.705860    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkqdq\" (UniqueName: \"kubernetes.io/projected/adb62d9b-f7b2-4118-8669-cda1e51557e6-kube-api-access-wkqdq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:43.705892 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.705887    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj8lq\" (UniqueName: \"kubernetes.io/projected/1aa65e87-cda6-468f-b669-44b5121ed498-kube-api-access-tj8lq\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:43.717437 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.717408    4282 generic.go:356] "Generic (PLEG): container finished" podID="1aa65e87-cda6-468f-b669-44b5121ed498" containerID="40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0" exitCode=0
Oct 30 22:34:43.717602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.717448    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"
Oct 30 22:34:43.717602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.717501    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz" event={"ID":"1aa65e87-cda6-468f-b669-44b5121ed498","Type":"ContainerDied","Data":"40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0"}
Oct 30 22:34:43.717602 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.717568    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz" event={"ID":"1aa65e87-cda6-468f-b669-44b5121ed498","Type":"ContainerDied","Data":"555ef1d9c999094ebf18fc69d4b677f62cc6db4a0e6a93b85e2fb98a127ff038"}
Oct 30 22:34:43.717780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.717603    4282 scope.go:122] "RemoveContainer" containerID="40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0"
Oct 30 22:34:43.718925 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.718902    4282 generic.go:356] "Generic (PLEG): container finished" podID="adb62d9b-f7b2-4118-8669-cda1e51557e6" containerID="1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7" exitCode=0
Oct 30 22:34:43.719068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.718941    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"
Oct 30 22:34:43.719068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.718998    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96" event={"ID":"adb62d9b-f7b2-4118-8669-cda1e51557e6","Type":"ContainerDied","Data":"1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7"}
Oct 30 22:34:43.719068 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.719048    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96" event={"ID":"adb62d9b-f7b2-4118-8669-cda1e51557e6","Type":"ContainerDied","Data":"19e856fcfd09e2aad76bbd691f5749439d2ae2d44919ffd8511a8228b3625c34"}
Oct 30 22:34:43.727656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.727639    4282 scope.go:122] "RemoveContainer" containerID="40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0"
Oct 30 22:34:43.727953 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:43.727934    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0\": container with ID starting with 40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0 not found: ID does not exist" containerID="40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0"
Oct 30 22:34:43.728044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.727961    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0"} err="failed to get container status \"40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0\": rpc error: code = NotFound desc = could not find container \"40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0\": container with ID starting with 40bca19f56b60d383975298ab572f992c970376f85246287516b90907326c5f0 not found: ID does not exist"
Oct 30 22:34:43.728044 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.727977    4282 scope.go:122] "RemoveContainer" containerID="1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7"
Oct 30 22:34:43.736595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.736576    4282 scope.go:122] "RemoveContainer" containerID="1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7"
Oct 30 22:34:43.736846 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:43.736830    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7\": container with ID starting with 1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7 not found: ID does not exist" containerID="1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7"
Oct 30 22:34:43.736944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.736853    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7"} err="failed to get container status \"1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7\": rpc error: code = NotFound desc = could not find container \"1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7\": container with ID starting with 1c9a5b4249fcaf8b38df7f88407260fc2c1c46c3271632c65ab8316a5e9464e7 not found: ID does not exist"
Oct 30 22:34:43.746427 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.746403    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"]
Oct 30 22:34:43.750768 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.750744    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a39158-75666cdd95-pbc96"]
Oct 30 22:34:43.759132 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.759109    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"]
Oct 30 22:34:43.762079 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:43.762060    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a39158-84bc5446f-r9dtz"]
Oct 30 22:34:45.290649 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:45.290618    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:34:45.462411 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:45.462376    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa65e87-cda6-468f-b669-44b5121ed498" path="/var/lib/kubelet/pods/1aa65e87-cda6-468f-b669-44b5121ed498/volumes"
Oct 30 22:34:45.462794 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:45.462779    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb62d9b-f7b2-4118-8669-cda1e51557e6" path="/var/lib/kubelet/pods/adb62d9b-f7b2-4118-8669-cda1e51557e6/volumes"
Oct 30 22:34:46.529962 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:46.529925    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl" podUID="b1503756-ef8b-4706-bdb1-d11f3dda8b56" containerName="payload-processing" containerID="cri-o://6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a" gracePeriod=25
Oct 30 22:34:49.057806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:49.057775    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl" podUID="b1503756-ef8b-4706-bdb1-d11f3dda8b56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:34:50.796116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:50.796084    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"]
Oct 30 22:34:50.796725 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:50.796289    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy" containerID="cri-o://d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b" gracePeriod=30
Oct 30 22:34:56.055901 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.055874    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:34:56.136227 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136140    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71d838db-a10d-4705-8c50-09e5ce26689a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d838db-a10d-4705-8c50-09e5ce26689a-istiod-ca-cert\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136258    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-socket\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136312    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-data\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136356    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-credential-socket\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136413 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136392    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-certs\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136491    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71d838db-a10d-4705-8c50-09e5ce26689a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d838db-a10d-4705-8c50-09e5ce26689a-istio-podinfo\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136541    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-kube-api-access-dndwd\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-kube-api-access-dndwd\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136589    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-istio-token\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-istio-token\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136623    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-socket" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:56.136703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136651    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-envoy\") pod \"71d838db-a10d-4705-8c50-09e5ce26689a\" (UID: \"71d838db-a10d-4705-8c50-09e5ce26689a\") "
Oct 30 22:34:56.136703 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136661    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-data" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:56.136999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136724    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-credential-socket" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:56.136999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.136781    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71d838db-a10d-4705-8c50-09e5ce26689a-istiod-ca-cert" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:56.137164 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.137096    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-certs" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:56.137401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.137163    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71d838db-a10d-4705-8c50-09e5ce26689a-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.137401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.137185    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.137401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.137200    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.137401 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.137214    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.139043 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.139008    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71d838db-a10d-4705-8c50-09e5ce26689a-istio-podinfo" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:56.139043 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.139030    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-envoy" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:56.139422 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.139398    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-istio-token" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:56.139557 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.139428    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-kube-api-access-dndwd" pod "71d838db-a10d-4705-8c50-09e5ce26689a" (UID: "71d838db-a10d-4705-8c50-09e5ce26689a"). InnerVolumeSpecName "kube-api-access-dndwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:56.237930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.237903    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.237930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.237927    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71d838db-a10d-4705-8c50-09e5ce26689a-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.238100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.237936    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndwd\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-kube-api-access-dndwd\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.238100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.237946    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71d838db-a10d-4705-8c50-09e5ce26689a-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.238100 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.237955    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71d838db-a10d-4705-8c50-09e5ce26689a-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:56.789609 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.789575    4282 generic.go:356] "Generic (PLEG): container finished" podID="71d838db-a10d-4705-8c50-09e5ce26689a" containerID="d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b" exitCode=0
Oct 30 22:34:56.789839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.789633    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"
Oct 30 22:34:56.789839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.789667    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" event={"ID":"71d838db-a10d-4705-8c50-09e5ce26689a","Type":"ContainerDied","Data":"d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b"}
Oct 30 22:34:56.789839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.789738    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp" event={"ID":"71d838db-a10d-4705-8c50-09e5ce26689a","Type":"ContainerDied","Data":"c2148ca1dd9598226578c98c2833ec2d4f013e998338aa00e3d7b93d7cb8e2c5"}
Oct 30 22:34:56.789839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.789768    4282 scope.go:122] "RemoveContainer" containerID="d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b"
Oct 30 22:34:56.799485 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.799451    4282 scope.go:122] "RemoveContainer" containerID="92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f"
Oct 30 22:34:56.807687 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.807670    4282 scope.go:122] "RemoveContainer" containerID="d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b"
Oct 30 22:34:56.807937 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:56.807919    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b\": container with ID starting with d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b not found: ID does not exist" containerID="d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b"
Oct 30 22:34:56.807995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.807946    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b"} err="failed to get container status \"d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b\": rpc error: code = NotFound desc = could not find container \"d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b\": container with ID starting with d63c27cba2befe0aa3e61695c91146d2dcff425c48a39a2541e2629c361af82b not found: ID does not exist"
Oct 30 22:34:56.807995 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.807965    4282 scope.go:122] "RemoveContainer" containerID="92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f"
Oct 30 22:34:56.808202 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:34:56.808185    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f\": container with ID starting with 92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f not found: ID does not exist" containerID="92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f"
Oct 30 22:34:56.808244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.808209    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f"} err="failed to get container status \"92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f\": rpc error: code = NotFound desc = could not find container \"92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f\": container with ID starting with 92c37465afafac6cd05b7208ff4d30f0f85e1bfa0a9ecf1be50420c9ef4a3b8f not found: ID does not exist"
Oct 30 22:34:56.812032 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.812010    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"]
Oct 30 22:34:56.817868 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:56.817841    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-26a128-openshift-default-67965c7d7c-rrzcp"]
Oct 30 22:34:57.463137 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:57.463103    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" path="/var/lib/kubelet/pods/71d838db-a10d-4705-8c50-09e5ce26689a/volumes"
Oct 30 22:34:59.057689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:34:59.057651    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl" podUID="b1503756-ef8b-4706-bdb1-d11f3dda8b56" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:35:01.463916 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:01.463883    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"]
Oct 30 22:35:01.665396 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:01.665360    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"]
Oct 30 22:35:01.665671 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:01.665624    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7" podUID="fdb43687-6eea-4cf8-98ac-1810e4d72e9d" containerName="maas-api" containerID="cri-o://9591ffb84a27ae36758e5d6380794238c41a5fd9c09b6c786d3f3a916da35f14" gracePeriod=30
Oct 30 22:35:01.823281 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:01.823251    4282 generic.go:356] "Generic (PLEG): container finished" podID="fdb43687-6eea-4cf8-98ac-1810e4d72e9d" containerID="9591ffb84a27ae36758e5d6380794238c41a5fd9c09b6c786d3f3a916da35f14" exitCode=0
Oct 30 22:35:01.823440 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:01.823351    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7" event={"ID":"fdb43687-6eea-4cf8-98ac-1810e4d72e9d","Type":"ContainerDied","Data":"9591ffb84a27ae36758e5d6380794238c41a5fd9c09b6c786d3f3a916da35f14"}
Oct 30 22:35:01.938601 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:01.938574    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:35:02.100182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.100096    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-kube-api-access-9zklp\" (UniqueName: \"kubernetes.io/projected/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-kube-api-access-9zklp\") pod \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") "
Oct 30 22:35:02.100182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.100176    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-openshift-service-ca\") pod \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") "
Oct 30 22:35:02.100454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.100230    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-maas-api-tls\") pod \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") "
Oct 30 22:35:02.100454 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.100324    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-metrics-certs\") pod \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\" (UID: \"fdb43687-6eea-4cf8-98ac-1810e4d72e9d\") "
Oct 30 22:35:02.100607 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.100560    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-openshift-service-ca" pod "fdb43687-6eea-4cf8-98ac-1810e4d72e9d" (UID: "fdb43687-6eea-4cf8-98ac-1810e4d72e9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:02.100712 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.100692    4282 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-openshift-service-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:02.102462 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.102439    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-metrics-certs" pod "fdb43687-6eea-4cf8-98ac-1810e4d72e9d" (UID: "fdb43687-6eea-4cf8-98ac-1810e4d72e9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:02.102558 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.102489    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-maas-api-tls" pod "fdb43687-6eea-4cf8-98ac-1810e4d72e9d" (UID: "fdb43687-6eea-4cf8-98ac-1810e4d72e9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:02.102658 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.102639    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-kube-api-access-9zklp" pod "fdb43687-6eea-4cf8-98ac-1810e4d72e9d" (UID: "fdb43687-6eea-4cf8-98ac-1810e4d72e9d"). InnerVolumeSpecName "kube-api-access-9zklp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:02.201316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.201282    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zklp\" (UniqueName: \"kubernetes.io/projected/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-kube-api-access-9zklp\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:02.201316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.201312    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-maas-api-tls\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:02.201316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.201322    4282 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdb43687-6eea-4cf8-98ac-1810e4d72e9d-metrics-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:02.831909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.831866    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7" event={"ID":"fdb43687-6eea-4cf8-98ac-1810e4d72e9d","Type":"ContainerDied","Data":"5a7e97fe50602360fd22eda82ea0d966ed694ce157e1f03b4dce8c8a88048f60"}
Oct 30 22:35:02.831909 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.831900    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"
Oct 30 22:35:02.832416 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.831928    4282 scope.go:122] "RemoveContainer" containerID="9591ffb84a27ae36758e5d6380794238c41a5fd9c09b6c786d3f3a916da35f14"
Oct 30 22:35:02.854655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.854633    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"]
Oct 30 22:35:02.860160 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:02.860137    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-18ba2f-59fcf4685c-nlnn7"]
Oct 30 22:35:03.464589 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:03.464555    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb43687-6eea-4cf8-98ac-1810e4d72e9d" path="/var/lib/kubelet/pods/fdb43687-6eea-4cf8-98ac-1810e4d72e9d/volumes"
Oct 30 22:35:04.496304 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:04.496266    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:35:04.649024 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:04.648988    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:35:06.464951 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.464899    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx" podUID="58e5a72d-e1d9-4216-af8d-1131d1afe450" containerName="payload-pre-processing" containerID="cri-o://90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8" gracePeriod=25
Oct 30 22:35:06.782145 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.782121    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:35:06.854447 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.854417    4282 generic.go:356] "Generic (PLEG): container finished" podID="b1503756-ef8b-4706-bdb1-d11f3dda8b56" containerID="6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a" exitCode=0
Oct 30 22:35:06.854668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.854457    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"
Oct 30 22:35:06.854668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.854523    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl" event={"ID":"b1503756-ef8b-4706-bdb1-d11f3dda8b56","Type":"ContainerDied","Data":"6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a"}
Oct 30 22:35:06.854668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.854590    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl" event={"ID":"b1503756-ef8b-4706-bdb1-d11f3dda8b56","Type":"ContainerDied","Data":"6f4f1410c82f95faf75fbbc4e33b97676a2c390c2caac4b54017531e1d22243b"}
Oct 30 22:35:06.854668 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.854623    4282 scope.go:122] "RemoveContainer" containerID="6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a"
Oct 30 22:35:06.863821 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.863803    4282 scope.go:122] "RemoveContainer" containerID="6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a"
Oct 30 22:35:06.864107 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:06.864084    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a\": container with ID starting with 6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a not found: ID does not exist" containerID="6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a"
Oct 30 22:35:06.864220 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.864115    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a"} err="failed to get container status \"6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a\": rpc error: code = NotFound desc = could not find container \"6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a\": container with ID starting with 6adae4ef167b7ac466aae2581bdd0406ff9cf1d63071633e91b3345ad803b55a not found: ID does not exist"
Oct 30 22:35:06.950965 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.950938    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1503756-ef8b-4706-bdb1-d11f3dda8b56-kube-api-access-rkphz\" (UniqueName: \"kubernetes.io/projected/b1503756-ef8b-4706-bdb1-d11f3dda8b56-kube-api-access-rkphz\") pod \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\" (UID: \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\") "
Oct 30 22:35:06.951111 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.950988    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1503756-ef8b-4706-bdb1-d11f3dda8b56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1503756-ef8b-4706-bdb1-d11f3dda8b56-plugins-config-volume\") pod \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\" (UID: \"b1503756-ef8b-4706-bdb1-d11f3dda8b56\") "
Oct 30 22:35:06.951349 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.951330    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1503756-ef8b-4706-bdb1-d11f3dda8b56-plugins-config-volume" pod "b1503756-ef8b-4706-bdb1-d11f3dda8b56" (UID: "b1503756-ef8b-4706-bdb1-d11f3dda8b56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:06.953182 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:06.953156    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1503756-ef8b-4706-bdb1-d11f3dda8b56-kube-api-access-rkphz" pod "b1503756-ef8b-4706-bdb1-d11f3dda8b56" (UID: "b1503756-ef8b-4706-bdb1-d11f3dda8b56"). InnerVolumeSpecName "kube-api-access-rkphz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:07.052754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:07.052696    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkphz\" (UniqueName: \"kubernetes.io/projected/b1503756-ef8b-4706-bdb1-d11f3dda8b56-kube-api-access-rkphz\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:07.052754 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:07.052725    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1503756-ef8b-4706-bdb1-d11f3dda8b56-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:07.175364 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:07.175337    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"]
Oct 30 22:35:07.181812 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:07.181781    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-26a128-6c48654f-vldtl"]
Oct 30 22:35:07.462616 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:07.462583    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1503756-ef8b-4706-bdb1-d11f3dda8b56" path="/var/lib/kubelet/pods/b1503756-ef8b-4706-bdb1-d11f3dda8b56/volumes"
Oct 30 22:35:09.813330 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:09.813296    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"]
Oct 30 22:35:09.813834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:09.813549    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy" containerID="cri-o://d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8" gracePeriod=30
Oct 30 22:35:14.656143 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:14.656108    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx" podUID="58e5a72d-e1d9-4216-af8d-1131d1afe450" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:35:15.067670 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.067643    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:35:15.141591 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141560    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-socket\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.141763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141608    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-data\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.141763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141634    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-credential-socket\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.141763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141654    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-certs\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.141763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141676    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-envoy\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.141763 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141749    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746fa886-2c5e-4eca-98f9-734076279325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746fa886-2c5e-4eca-98f9-734076279325-istiod-ca-cert\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.142117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141787    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/746fa886-2c5e-4eca-98f9-734076279325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746fa886-2c5e-4eca-98f9-734076279325-istio-podinfo\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.142117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141810    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-socket" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:15.142117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141919    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-istio-token\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-istio-token\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.142117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141964    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-certs" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:15.142117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141975    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-data" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:15.142117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141990    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-kube-api-access-qqm8z\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-kube-api-access-qqm8z\") pod \"746fa886-2c5e-4eca-98f9-734076279325\" (UID: \"746fa886-2c5e-4eca-98f9-734076279325\") "
Oct 30 22:35:15.142117 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.141993    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-credential-socket" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:15.142449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.142201    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746fa886-2c5e-4eca-98f9-734076279325-istiod-ca-cert" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:15.142449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.142344    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.142449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.142367    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.142449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.142381    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.142449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.142394    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.142449 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.142406    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/746fa886-2c5e-4eca-98f9-734076279325-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.144166 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.144145    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-envoy" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:15.144251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.144238    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/746fa886-2c5e-4eca-98f9-734076279325-istio-podinfo" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:35:15.144677 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.144661    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-kube-api-access-qqm8z" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "kube-api-access-qqm8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:15.144727 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.144689    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-istio-token" pod "746fa886-2c5e-4eca-98f9-734076279325" (UID: "746fa886-2c5e-4eca-98f9-734076279325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:15.243923 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.243850    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/746fa886-2c5e-4eca-98f9-734076279325-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.243923 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.243877    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/746fa886-2c5e-4eca-98f9-734076279325-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.243923 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.243887    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.243923 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.243897    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqm8z\" (UniqueName: \"kubernetes.io/projected/746fa886-2c5e-4eca-98f9-734076279325-kube-api-access-qqm8z\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:15.907189 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.907153    4282 generic.go:356] "Generic (PLEG): container finished" podID="746fa886-2c5e-4eca-98f9-734076279325" containerID="d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8" exitCode=0
Oct 30 22:35:15.907704 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.907207    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"
Oct 30 22:35:15.907704 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.907211    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" event={"ID":"746fa886-2c5e-4eca-98f9-734076279325","Type":"ContainerDied","Data":"d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8"}
Oct 30 22:35:15.907704 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.907344    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz" event={"ID":"746fa886-2c5e-4eca-98f9-734076279325","Type":"ContainerDied","Data":"ac871aa10d100397ae058e29179951a4b8a7374c0ba9d4873b2baad90d9e2322"}
Oct 30 22:35:15.907704 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.907377    4282 scope.go:122] "RemoveContainer" containerID="d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8"
Oct 30 22:35:15.916540 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.916525    4282 scope.go:122] "RemoveContainer" containerID="d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8"
Oct 30 22:35:15.916812 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:15.916793    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8\": container with ID starting with d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8 not found: ID does not exist" containerID="d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8"
Oct 30 22:35:15.916887 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.916823    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8"} err="failed to get container status \"d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8\": rpc error: code = NotFound desc = could not find container \"d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8\": container with ID starting with d5ac83a116439a3e33f80d9dc259742b2fddcc4ceebe40e5dbb669b1c63d66e8 not found: ID does not exist"
Oct 30 22:35:15.925931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.925911    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"]
Oct 30 22:35:15.934317 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:15.934297    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-18ba2f-openshift-default-7866c74fb7-n4dgz"]
Oct 30 22:35:17.462108 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:17.462068    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746fa886-2c5e-4eca-98f9-734076279325" path="/var/lib/kubelet/pods/746fa886-2c5e-4eca-98f9-734076279325/volumes"
Oct 30 22:35:20.624448 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:20.624408    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"]
Oct 30 22:35:24.102170 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:24.102128    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:35:24.656715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:24.656678    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx" podUID="58e5a72d-e1d9-4216-af8d-1131d1afe450" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:35:25.624881 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:25.624839    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c" podUID="fbca5f55-1e74-457f-8f10-01e4601e8566" containerName="payload-processing" containerID="cri-o://a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821" gracePeriod=25
Oct 30 22:35:26.722930 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.722867    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:35:26.859206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.859179    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58e5a72d-e1d9-4216-af8d-1131d1afe450-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e5a72d-e1d9-4216-af8d-1131d1afe450-plugins-config-volume\") pod \"58e5a72d-e1d9-4216-af8d-1131d1afe450\" (UID: \"58e5a72d-e1d9-4216-af8d-1131d1afe450\") "
Oct 30 22:35:26.859403 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.859228    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58e5a72d-e1d9-4216-af8d-1131d1afe450-kube-api-access-hqw9b\" (UniqueName: \"kubernetes.io/projected/58e5a72d-e1d9-4216-af8d-1131d1afe450-kube-api-access-hqw9b\") pod \"58e5a72d-e1d9-4216-af8d-1131d1afe450\" (UID: \"58e5a72d-e1d9-4216-af8d-1131d1afe450\") "
Oct 30 22:35:26.859595 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.859564    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e5a72d-e1d9-4216-af8d-1131d1afe450-plugins-config-volume" pod "58e5a72d-e1d9-4216-af8d-1131d1afe450" (UID: "58e5a72d-e1d9-4216-af8d-1131d1afe450"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:26.861844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.861818    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58e5a72d-e1d9-4216-af8d-1131d1afe450-kube-api-access-hqw9b" pod "58e5a72d-e1d9-4216-af8d-1131d1afe450" (UID: "58e5a72d-e1d9-4216-af8d-1131d1afe450"). InnerVolumeSpecName "kube-api-access-hqw9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:26.960790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.960759    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58e5a72d-e1d9-4216-af8d-1131d1afe450-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:26.960790 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.960781    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqw9b\" (UniqueName: \"kubernetes.io/projected/58e5a72d-e1d9-4216-af8d-1131d1afe450-kube-api-access-hqw9b\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:26.961826 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.961801    4282 generic.go:356] "Generic (PLEG): container finished" podID="58e5a72d-e1d9-4216-af8d-1131d1afe450" containerID="90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8" exitCode=0
Oct 30 22:35:26.961949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.961850    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"
Oct 30 22:35:26.961949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.961850    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx" event={"ID":"58e5a72d-e1d9-4216-af8d-1131d1afe450","Type":"ContainerDied","Data":"90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8"}
Oct 30 22:35:26.962034 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.961980    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx" event={"ID":"58e5a72d-e1d9-4216-af8d-1131d1afe450","Type":"ContainerDied","Data":"3fb989a37c4a70c07ecaa71012d7f1e88350f199415c52501393b4642e3bf7be"}
Oct 30 22:35:26.962034 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.962011    4282 scope.go:122] "RemoveContainer" containerID="90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8"
Oct 30 22:35:26.971127 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.971112    4282 scope.go:122] "RemoveContainer" containerID="90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8"
Oct 30 22:35:26.971376 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:26.971360    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8\": container with ID starting with 90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8 not found: ID does not exist" containerID="90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8"
Oct 30 22:35:26.971442 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.971387    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8"} err="failed to get container status \"90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8\": rpc error: code = NotFound desc = could not find container \"90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8\": container with ID starting with 90246f3814866b5123c83371f1cf073ca0df9edb1454264a31adf4dc66983ec8 not found: ID does not exist"
Oct 30 22:35:26.982941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.982883    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"]
Oct 30 22:35:26.987251 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:26.987229    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-18ba2f-d456d4d5f-s7jqx"]
Oct 30 22:35:27.462085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:27.462055    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58e5a72d-e1d9-4216-af8d-1131d1afe450" path="/var/lib/kubelet/pods/58e5a72d-e1d9-4216-af8d-1131d1afe450/volumes"
Oct 30 22:35:28.502008 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:28.501975    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c" podUID="fbca5f55-1e74-457f-8f10-01e4601e8566" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:35:29.377903 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:29.377865    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"]
Oct 30 22:35:29.378177 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:29.378143    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy" containerID="cri-o://95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a" gracePeriod=30
Oct 30 22:35:34.625739 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.625696    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:35:34.648568 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:34.648494    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:35:34.737248 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737220    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b99d70de-0e2a-4574-b914-a667183840f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99d70de-0e2a-4574-b914-a667183840f6-istio-podinfo\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737261    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99d70de-0e2a-4574-b914-a667183840f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99d70de-0e2a-4574-b914-a667183840f6-istiod-ca-cert\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737313    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-data\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737377    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-istio-token\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-istio-token\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737409    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-envoy\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737458    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-socket\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737529 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737511    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-kube-api-access-vpcff\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-kube-api-access-vpcff\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737857 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737559    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-credential-socket\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737857 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737605    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-certs\") pod \"b99d70de-0e2a-4574-b914-a667183840f6\" (UID: \"b99d70de-0e2a-4574-b914-a667183840f6\") "
Oct 30 22:35:34.737857 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737720    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-data" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:34.737857 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737835    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-socket" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:34.738069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.737932    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-credential-socket" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:34.738069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.738046    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-certs" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:34.738069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.738057    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99d70de-0e2a-4574-b914-a667183840f6-istiod-ca-cert" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:34.738213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.738153    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.738213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.738192    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.738213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.738210    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.738341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.738225    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.740002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.739973    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-kube-api-access-vpcff" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "kube-api-access-vpcff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:34.740002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.739985    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-envoy" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:34.740372 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.740348    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b99d70de-0e2a-4574-b914-a667183840f6-istio-podinfo" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:35:34.740513 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.740370    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-istio-token" pod "b99d70de-0e2a-4574-b914-a667183840f6" (UID: "b99d70de-0e2a-4574-b914-a667183840f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:34.839381 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.839348    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b99d70de-0e2a-4574-b914-a667183840f6-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.839381 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.839375    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b99d70de-0e2a-4574-b914-a667183840f6-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.839381 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.839384    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.839656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.839394    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b99d70de-0e2a-4574-b914-a667183840f6-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.839656 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.839403    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpcff\" (UniqueName: \"kubernetes.io/projected/b99d70de-0e2a-4574-b914-a667183840f6-kube-api-access-vpcff\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:34.951520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:34.951487    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"]
Oct 30 22:35:35.004516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.004452    4282 generic.go:356] "Generic (PLEG): container finished" podID="b99d70de-0e2a-4574-b914-a667183840f6" containerID="95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a" exitCode=0
Oct 30 22:35:35.004681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.004539    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" event={"ID":"b99d70de-0e2a-4574-b914-a667183840f6","Type":"ContainerDied","Data":"95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a"}
Oct 30 22:35:35.004681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.004631    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn" event={"ID":"b99d70de-0e2a-4574-b914-a667183840f6","Type":"ContainerDied","Data":"9beae7ceaca96ad1581bf597fc2a03fdc400f10926d44afbbf6564eec9d456f9"}
Oct 30 22:35:35.004681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.004642    4282 scope.go:122] "RemoveContainer" containerID="95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a"
Oct 30 22:35:35.004681 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.004549    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"
Oct 30 22:35:35.015175 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.015155    4282 scope.go:122] "RemoveContainer" containerID="282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c"
Oct 30 22:35:35.023856 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.023838    4282 scope.go:122] "RemoveContainer" containerID="95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a"
Oct 30 22:35:35.024147 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:35.024130    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a\": container with ID starting with 95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a not found: ID does not exist" containerID="95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a"
Oct 30 22:35:35.024198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.024155    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a"} err="failed to get container status \"95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a\": rpc error: code = NotFound desc = could not find container \"95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a\": container with ID starting with 95ba646183fb38ee34501248abdd895f97f55524e3fef45d929ba0fa876d616a not found: ID does not exist"
Oct 30 22:35:35.024198 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.024171    4282 scope.go:122] "RemoveContainer" containerID="282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c"
Oct 30 22:35:35.024427 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:35.024410    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c\": container with ID starting with 282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c not found: ID does not exist" containerID="282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c"
Oct 30 22:35:35.024563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.024430    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c"} err="failed to get container status \"282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c\": rpc error: code = NotFound desc = could not find container \"282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c\": container with ID starting with 282ca983b09c1eade5c3e21e2e996e3c83a8d1a1a632e0d73cbf8e04c48e891c not found: ID does not exist"
Oct 30 22:35:35.039841 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.039820    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"]
Oct 30 22:35:35.055002 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.054972    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-38d6e2-openshift-default-dbbfc47b9-jtstn"]
Oct 30 22:35:35.463820 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:35.463781    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99d70de-0e2a-4574-b914-a667183840f6" path="/var/lib/kubelet/pods/b99d70de-0e2a-4574-b914-a667183840f6/volumes"
Oct 30 22:35:38.501242 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:38.501203    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c" podUID="fbca5f55-1e74-457f-8f10-01e4601e8566" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:35:40.280167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.280132    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"]
Oct 30 22:35:40.281325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281292    4282 cpu_manager.go:414] "containerMap: removing container" podUID="7760b38b-dcfa-463f-969a-954a60353919" containerName="maas-api"
Oct 30 22:35:40.281325 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281324    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7760b38b-dcfa-463f-969a-954a60353919" containerName="maas-api"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281342    4282 cpu_manager.go:414] "containerMap: removing container" podUID="cc889df4-c49f-4b23-9db5-2561028d5b2e" containerName="payload-pre-processing"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281350    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc889df4-c49f-4b23-9db5-2561028d5b2e" containerName="payload-pre-processing"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281364    4282 cpu_manager.go:414] "containerMap: removing container" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281371    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281384    4282 cpu_manager.go:414] "containerMap: removing container" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281391    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281411    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281418    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281427    4282 cpu_manager.go:414] "containerMap: removing container" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281434    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281444    4282 cpu_manager.go:414] "containerMap: removing container" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281451    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281461    4282 cpu_manager.go:414] "containerMap: removing container" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281488    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281498    4282 cpu_manager.go:414] "containerMap: removing container" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281505    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281518    4282 cpu_manager.go:414] "containerMap: removing container" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281524    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281532    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b1503756-ef8b-4706-bdb1-d11f3dda8b56" containerName="payload-processing"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281541    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1503756-ef8b-4706-bdb1-d11f3dda8b56" containerName="payload-processing"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281551    4282 cpu_manager.go:414] "containerMap: removing container" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281557    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281577    4282 cpu_manager.go:414] "containerMap: removing container" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281583    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281593    4282 cpu_manager.go:414] "containerMap: removing container" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281600    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281608    4282 cpu_manager.go:414] "containerMap: removing container" podUID="bf37c164-c4c7-4e73-b615-bb1b9ae62dea" containerName="payload-processing"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281615    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf37c164-c4c7-4e73-b615-bb1b9ae62dea" containerName="payload-processing"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281625    4282 cpu_manager.go:414] "containerMap: removing container" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy"
Oct 30 22:35:40.281612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281632    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281663    4282 cpu_manager.go:414] "containerMap: removing container" podUID="fdb43687-6eea-4cf8-98ac-1810e4d72e9d" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281672    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb43687-6eea-4cf8-98ac-1810e4d72e9d" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281684    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281690    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281724    4282 cpu_manager.go:414] "containerMap: removing container" podUID="07b0d53f-2a75-460e-89f9-a8c2a03608de" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281733    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b0d53f-2a75-460e-89f9-a8c2a03608de" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281749    4282 cpu_manager.go:414] "containerMap: removing container" podUID="9fd149ee-426a-4670-b647-02f50aef6219" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281757    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fd149ee-426a-4670-b647-02f50aef6219" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281773    4282 cpu_manager.go:414] "containerMap: removing container" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281781    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281792    4282 cpu_manager.go:414] "containerMap: removing container" podUID="fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281799    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281810    4282 cpu_manager.go:414] "containerMap: removing container" podUID="caa2f586-028f-4202-8891-5fe93c3650d2" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281816    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa2f586-028f-4202-8891-5fe93c3650d2" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281825    4282 cpu_manager.go:414] "containerMap: removing container" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281833    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281845    4282 cpu_manager.go:414] "containerMap: removing container" podUID="adb62d9b-f7b2-4118-8669-cda1e51557e6" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281851    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb62d9b-f7b2-4118-8669-cda1e51557e6" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281865    4282 cpu_manager.go:414] "containerMap: removing container" podUID="e94ea251-952e-4a62-bb7d-c39f389542ee" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281871    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94ea251-952e-4a62-bb7d-c39f389542ee" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281881    4282 cpu_manager.go:414] "containerMap: removing container" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281887    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281898    4282 cpu_manager.go:414] "containerMap: removing container" podUID="2486a5ab-245c-40cb-93d0-c01ab6408012" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281905    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2486a5ab-245c-40cb-93d0-c01ab6408012" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281915    4282 cpu_manager.go:414] "containerMap: removing container" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281922    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281937    4282 cpu_manager.go:414] "containerMap: removing container" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281944    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281950    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281955    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281967    4282 cpu_manager.go:414] "containerMap: removing container" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281973    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281986    4282 cpu_manager.go:414] "containerMap: removing container" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.281992    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282001    4282 cpu_manager.go:414] "containerMap: removing container" podUID="84b39bf1-07b7-46c4-bcf7-3b955a955bd8" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282009    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b39bf1-07b7-46c4-bcf7-3b955a955bd8" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282020    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282027    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282034    4282 cpu_manager.go:414] "containerMap: removing container" podUID="58e5a72d-e1d9-4216-af8d-1131d1afe450" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282039    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58e5a72d-e1d9-4216-af8d-1131d1afe450" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282045    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282049    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282056    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282060    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282066    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f7568c7b-2b8f-41d2-8554-e33f16cbf42f" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282070    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7568c7b-2b8f-41d2-8554-e33f16cbf42f" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282077    4282 cpu_manager.go:414] "containerMap: removing container" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282081    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282088    4282 cpu_manager.go:414] "containerMap: removing container" podUID="1be12dee-de32-4f71-baad-46d448718612" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282092    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1be12dee-de32-4f71-baad-46d448718612" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282099    4282 cpu_manager.go:414] "containerMap: removing container" podUID="a1a3ae17-f62e-4b49-83a6-6216fd549636" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282103    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a3ae17-f62e-4b49-83a6-6216fd549636" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282110    4282 cpu_manager.go:414] "containerMap: removing container" podUID="66680916-61a4-451c-b6d8-42a57d23705d" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282114    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66680916-61a4-451c-b6d8-42a57d23705d" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282128    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0688b402-ba16-4049-a38e-ece12d389c7d" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282132    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0688b402-ba16-4049-a38e-ece12d389c7d" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282137    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ae82a63f-9f4c-4a2a-b713-127e52277966" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282143    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae82a63f-9f4c-4a2a-b713-127e52277966" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282150    4282 cpu_manager.go:414] "containerMap: removing container" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282154    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282159    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282164    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282169    4282 cpu_manager.go:414] "containerMap: removing container" podUID="1aa65e87-cda6-468f-b669-44b5121ed498" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282173    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa65e87-cda6-468f-b669-44b5121ed498" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282180    4282 cpu_manager.go:414] "containerMap: removing container" podUID="cc88f186-1a26-4dde-a4b3-2cda4e3f218e" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282187    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc88f186-1a26-4dde-a4b3-2cda4e3f218e" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282196    4282 cpu_manager.go:414] "containerMap: removing container" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282202    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282209    4282 cpu_manager.go:414] "containerMap: removing container" podUID="21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282213    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282219    4282 cpu_manager.go:414] "containerMap: removing container" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282222    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282230    4282 cpu_manager.go:414] "containerMap: removing container" podUID="31d7feb5-8074-4993-ae62-5352da049922" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282234    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d7feb5-8074-4993-ae62-5352da049922" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282240    4282 cpu_manager.go:414] "containerMap: removing container" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282244    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282250    4282 cpu_manager.go:414] "containerMap: removing container" podUID="3900d25b-7d5e-4b35-826c-afaec7394e5b" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282254    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3900d25b-7d5e-4b35-826c-afaec7394e5b" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282263    4282 cpu_manager.go:414] "containerMap: removing container" podUID="5b7ec276-992d-40e4-a83e-65b55a7f7df7" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282270    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b7ec276-992d-40e4-a83e-65b55a7f7df7" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282278    4282 cpu_manager.go:414] "containerMap: removing container" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282283    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282291    4282 cpu_manager.go:414] "containerMap: removing container" podUID="0438efc5-012a-4fb4-98cf-7b8b9af86145" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282295    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0438efc5-012a-4fb4-98cf-7b8b9af86145" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282305    4282 cpu_manager.go:414] "containerMap: removing container" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282310    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282317    4282 cpu_manager.go:414] "containerMap: removing container" podUID="303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282322    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282328    4282 cpu_manager.go:414] "containerMap: removing container" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282332    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282419    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="8410d5b0-f783-412b-8075-f83b710df94e" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282438    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbedb77e-2f7a-4424-8e14-04e9f5b4cdfc" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282448    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1503756-ef8b-4706-bdb1-d11f3dda8b56" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282458    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="19855a1c-4f0c-478c-acf4-8eb39097fc92" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282487    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282497    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf37c164-c4c7-4e73-b615-bb1b9ae62dea" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282504    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="be9d37c2-8e42-411e-8e26-85a048947be6" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282515    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282524    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb43687-6eea-4cf8-98ac-1810e4d72e9d" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282531    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282538    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b0d53f-2a75-460e-89f9-a8c2a03608de" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282548    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="28591d5d-928e-407f-b292-81ce32dc3393" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282555    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fd149ee-426a-4670-b647-02f50aef6219" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282565    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282575    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43ffb1e-2508-4081-90e3-69febec2c84f" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282585    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ea8884-75a7-4608-b491-40cb633aca64" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282595    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb5e5a8-022a-4076-a0c4-4afc2f336c1e" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282603    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa2f586-028f-4202-8891-5fe93c3650d2" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282611    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="746fa886-2c5e-4eca-98f9-734076279325" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282618    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb62d9b-f7b2-4118-8669-cda1e51557e6" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282627    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94ea251-952e-4a62-bb7d-c39f389542ee" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282637    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e43d9d-1cf7-418f-a158-1a2d002dafcc" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282647    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="2486a5ab-245c-40cb-93d0-c01ab6408012" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282656    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="122c7ccb-de31-4bba-9b54-bbc4ad6a32c8" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282666    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="deecd927-414a-41ec-9341-89cc0af48c7f" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282673    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2041356-82e2-4aaf-bf87-0ac90df55ba1" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282681    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9cd784-b387-42ba-b0d3-c0a3c2dadc7d" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282693    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c56d6f-0193-4922-a60f-b0aa5a71910b" containerName="cleanup"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282701    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282709    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282717    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91cee75-83ba-468a-a4c0-9c40820adedc" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282725    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b39bf1-07b7-46c4-bcf7-3b955a955bd8" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282734    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7e3af0-2a59-4f1b-bc9c-7530aec5ae75" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282745    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="58e5a72d-e1d9-4216-af8d-1131d1afe450" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282753    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282761    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a563a277-f347-4a67-b2a7-854282b5504a" containerName="main"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282769    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7568c7b-2b8f-41d2-8554-e33f16cbf42f" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282779    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282790    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="716883c7-6993-4ea2-b694-f334aba562c0" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282800    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1be12dee-de32-4f71-baad-46d448718612" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282808    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a3ae17-f62e-4b49-83a6-6216fd549636" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282817    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="66680916-61a4-451c-b6d8-42a57d23705d" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282827    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0688b402-ba16-4049-a38e-ece12d389c7d" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282837    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae82a63f-9f4c-4a2a-b713-127e52277966" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282846    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b416f91-5c95-43cd-bdc6-402bca298e02" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282854    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6d46316-4736-474f-aa55-71da33c19818" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282862    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa65e87-cda6-468f-b669-44b5121ed498" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282871    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc88f186-1a26-4dde-a4b3-2cda4e3f218e" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282880    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="54455775-f1d8-4e14-afe1-fd8ad99fb188" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282889    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a8d7f4-240c-4a9c-bdf8-b0bdd193d8cf" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282897    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc6b7c6-166a-4d5f-8da3-ee7345189759" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282907    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d7feb5-8074-4993-ae62-5352da049922" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282917    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="3900d25b-7d5e-4b35-826c-afaec7394e5b" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282924    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b7ec276-992d-40e4-a83e-65b55a7f7df7" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282932    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a61cf4-e93a-4fad-a160-e6f385fcf88f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282943    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="0438efc5-012a-4fb4-98cf-7b8b9af86145" containerName="payload-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282955    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d6edcc-0583-4a8b-ac74-9e4bcc4501a3" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282964    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="303a8976-dc91-43cf-ae4e-4e4cfcd2f8aa" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282973    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b514a9-b9c4-4705-a789-a7acaba6a27f" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282985    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="7760b38b-dcfa-463f-969a-954a60353919" containerName="maas-api"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.282994    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc889df4-c49f-4b23-9db5-2561028d5b2e" containerName="payload-pre-processing"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.283004    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.283012    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99d70de-0e2a-4574-b914-a667183840f6" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.283021    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4440c1-907a-4752-9a69-197dfcb5ead4" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.283031    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="774525b8-c126-4535-a248-0b63d68e599a" containerName="istio-proxy"
Oct 30 22:35:40.282985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.283039    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="14311e13-e401-442b-814e-4aa4e751e23c" containerName="cleanup"
Oct 30 22:35:40.288346 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.288178    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.296120 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.296097    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"]
Oct 30 22:35:40.328003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.327974    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:35:40.328003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.327987    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\""
Oct 30 22:35:40.387521 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.387493    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.387660 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.387602    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkl7k\" (UniqueName: \"kubernetes.io/projected/ba418436-3047-4b2d-9efa-d0a8b9582aaa-kube-api-access-mkl7k\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.387704 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.387658    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.488876 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.488846    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkl7k\" (UniqueName: \"kubernetes.io/projected/ba418436-3047-4b2d-9efa-d0a8b9582aaa-kube-api-access-mkl7k\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.489054 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.488919    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.489054 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.488972    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.489750 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.489723    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.491516 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.491493    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.502931 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.502900    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkl7k\" (UniqueName: \"kubernetes.io/projected/ba418436-3047-4b2d-9efa-d0a8b9582aaa-kube-api-access-mkl7k\") pod \"maas-api-revoke-sub-da804873-5hzsj\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.609366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.609277    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:40.885738 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:40.885713    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"]
Oct 30 22:35:40.888253 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:35:40.888222    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba418436_3047_4b2d_9efa_d0a8b9582aaa.slice/crio-8b75b20f0c78969604b192ecc34624b8c33eee8bc9b633cf1203d4320a04769c WatchSource:0}: Error finding container 8b75b20f0c78969604b192ecc34624b8c33eee8bc9b633cf1203d4320a04769c: Status 404 returned error can't find the container with id 8b75b20f0c78969604b192ecc34624b8c33eee8bc9b633cf1203d4320a04769c
Oct 30 22:35:41.036840 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:41.036808    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj" event={"ID":"ba418436-3047-4b2d-9efa-d0a8b9582aaa","Type":"ContainerStarted","Data":"8b75b20f0c78969604b192ecc34624b8c33eee8bc9b633cf1203d4320a04769c"}
Oct 30 22:35:42.042844 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:42.042757    4282 generic.go:356] "Generic (PLEG): container finished" podID="ba418436-3047-4b2d-9efa-d0a8b9582aaa" containerID="595fae39bea1a724a3f6d3e605f873b497c69a0b0b0e093e562cd2eeaaf07e9f" exitCode=0
Oct 30 22:35:42.043225 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:42.042854    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj" event={"ID":"ba418436-3047-4b2d-9efa-d0a8b9582aaa","Type":"ContainerDied","Data":"595fae39bea1a724a3f6d3e605f873b497c69a0b0b0e093e562cd2eeaaf07e9f"}
Oct 30 22:35:43.184978 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.184955    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:43.217060 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.217035    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-xks-ca\") pod \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") "
Oct 30 22:35:43.217172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.217107    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-openshift-ca\") pod \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") "
Oct 30 22:35:43.217172 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.217164    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba418436-3047-4b2d-9efa-d0a8b9582aaa-kube-api-access-mkl7k\" (UniqueName: \"kubernetes.io/projected/ba418436-3047-4b2d-9efa-d0a8b9582aaa-kube-api-access-mkl7k\") pod \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\" (UID: \"ba418436-3047-4b2d-9efa-d0a8b9582aaa\") "
Oct 30 22:35:43.217443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.217422    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-openshift-ca" pod "ba418436-3047-4b2d-9efa-d0a8b9582aaa" (UID: "ba418436-3047-4b2d-9efa-d0a8b9582aaa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:43.217582 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.217561    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-openshift-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:43.219338 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.219315    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-xks-ca" pod "ba418436-3047-4b2d-9efa-d0a8b9582aaa" (UID: "ba418436-3047-4b2d-9efa-d0a8b9582aaa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:43.219457 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.219432    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba418436-3047-4b2d-9efa-d0a8b9582aaa-kube-api-access-mkl7k" pod "ba418436-3047-4b2d-9efa-d0a8b9582aaa" (UID: "ba418436-3047-4b2d-9efa-d0a8b9582aaa"). InnerVolumeSpecName "kube-api-access-mkl7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:43.318974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.318917    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkl7k\" (UniqueName: \"kubernetes.io/projected/ba418436-3047-4b2d-9efa-d0a8b9582aaa-kube-api-access-mkl7k\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:43.318974 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:43.318941    4282 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba418436-3047-4b2d-9efa-d0a8b9582aaa-maas-api-xks-ca\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:44.053461 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:44.053432    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj"
Oct 30 22:35:44.053655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:44.053485    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da804873-5hzsj" event={"ID":"ba418436-3047-4b2d-9efa-d0a8b9582aaa","Type":"ContainerDied","Data":"8b75b20f0c78969604b192ecc34624b8c33eee8bc9b633cf1203d4320a04769c"}
Oct 30 22:35:44.053655 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:44.053534    4282 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b75b20f0c78969604b192ecc34624b8c33eee8bc9b633cf1203d4320a04769c"
Oct 30 22:35:45.878385 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:45.878357    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:35:45.945316 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:45.945292    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbca5f55-1e74-457f-8f10-01e4601e8566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbca5f55-1e74-457f-8f10-01e4601e8566-plugins-config-volume\") pod \"fbca5f55-1e74-457f-8f10-01e4601e8566\" (UID: \"fbca5f55-1e74-457f-8f10-01e4601e8566\") "
Oct 30 22:35:45.945502 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:45.945326    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbca5f55-1e74-457f-8f10-01e4601e8566-kube-api-access-8bs44\" (UniqueName: \"kubernetes.io/projected/fbca5f55-1e74-457f-8f10-01e4601e8566-kube-api-access-8bs44\") pod \"fbca5f55-1e74-457f-8f10-01e4601e8566\" (UID: \"fbca5f55-1e74-457f-8f10-01e4601e8566\") "
Oct 30 22:35:45.945675 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:45.945654    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbca5f55-1e74-457f-8f10-01e4601e8566-plugins-config-volume" pod "fbca5f55-1e74-457f-8f10-01e4601e8566" (UID: "fbca5f55-1e74-457f-8f10-01e4601e8566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:45.947565 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:45.947544    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbca5f55-1e74-457f-8f10-01e4601e8566-kube-api-access-8bs44" pod "fbca5f55-1e74-457f-8f10-01e4601e8566" (UID: "fbca5f55-1e74-457f-8f10-01e4601e8566"). InnerVolumeSpecName "kube-api-access-8bs44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:46.046871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.046841    4282 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbca5f55-1e74-457f-8f10-01e4601e8566-plugins-config-volume\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:46.046871 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.046868    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bs44\" (UniqueName: \"kubernetes.io/projected/fbca5f55-1e74-457f-8f10-01e4601e8566-kube-api-access-8bs44\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:46.063017 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.062986    4282 generic.go:356] "Generic (PLEG): container finished" podID="fbca5f55-1e74-457f-8f10-01e4601e8566" containerID="a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821" exitCode=0
Oct 30 22:35:46.063163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.063032    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"
Oct 30 22:35:46.063163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.063073    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c" event={"ID":"fbca5f55-1e74-457f-8f10-01e4601e8566","Type":"ContainerDied","Data":"a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821"}
Oct 30 22:35:46.063163 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.063133    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c" event={"ID":"fbca5f55-1e74-457f-8f10-01e4601e8566","Type":"ContainerDied","Data":"7e7f205e5e09ecdd2f1da7abd74d415eaee21d43def91ab9d9789eaddbd4d171"}
Oct 30 22:35:46.063297 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.063164    4282 scope.go:122] "RemoveContainer" containerID="a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821"
Oct 30 22:35:46.072629 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.072612    4282 scope.go:122] "RemoveContainer" containerID="a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821"
Oct 30 22:35:46.072896 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:46.072876    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821\": container with ID starting with a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821 not found: ID does not exist" containerID="a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821"
Oct 30 22:35:46.072964 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.072909    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821"} err="failed to get container status \"a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821\": rpc error: code = NotFound desc = could not find container \"a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821\": container with ID starting with a05a1f2fa2cac69674e27cb83d71484abd07ec5ea454b0b6722e21ee17576821 not found: ID does not exist"
Oct 30 22:35:46.090518 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.090488    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"]
Oct 30 22:35:46.098373 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:46.098349    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-38d6e2-6757c99c9-hm48c"]
Oct 30 22:35:47.462203 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:47.462173    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbca5f55-1e74-457f-8f10-01e4601e8566" path="/var/lib/kubelet/pods/fbca5f55-1e74-457f-8f10-01e4601e8566/volumes"
Oct 30 22:35:49.971773 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:49.971733    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main" containerID="cri-o://ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04" gracePeriod=45
Oct 30 22:35:50.223206 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.223148    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:35:50.285180 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.285147    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c6d10d-2eea-4866-a833-59cb87ba67e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/16c6d10d-2eea-4866-a833-59cb87ba67e3-tls-certs\") pod \"16c6d10d-2eea-4866-a833-59cb87ba67e3\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") "
Oct 30 22:35:50.285380 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.285188    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-dshm\") pod \"16c6d10d-2eea-4866-a833-59cb87ba67e3\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") "
Oct 30 22:35:50.285380 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.285207    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-model-cache\") pod \"16c6d10d-2eea-4866-a833-59cb87ba67e3\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") "
Oct 30 22:35:50.285380 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.285273    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-tmp-dir\") pod \"16c6d10d-2eea-4866-a833-59cb87ba67e3\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") "
Oct 30 22:35:50.285380 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.285375    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c6d10d-2eea-4866-a833-59cb87ba67e3-kube-api-access-rfchk\" (UniqueName: \"kubernetes.io/projected/16c6d10d-2eea-4866-a833-59cb87ba67e3-kube-api-access-rfchk\") pod \"16c6d10d-2eea-4866-a833-59cb87ba67e3\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") "
Oct 30 22:35:50.286443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.285419    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-home\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-home\") pod \"16c6d10d-2eea-4866-a833-59cb87ba67e3\" (UID: \"16c6d10d-2eea-4866-a833-59cb87ba67e3\") "
Oct 30 22:35:50.286443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.285577    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-model-cache" pod "16c6d10d-2eea-4866-a833-59cb87ba67e3" (UID: "16c6d10d-2eea-4866-a833-59cb87ba67e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.286443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.285893    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-home" pod "16c6d10d-2eea-4866-a833-59cb87ba67e3" (UID: "16c6d10d-2eea-4866-a833-59cb87ba67e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.286443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.286179    4282 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-home\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:50.286443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.286201    4282 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-model-cache\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:50.286807 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.286677    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-tmp-dir" pod "16c6d10d-2eea-4866-a833-59cb87ba67e3" (UID: "16c6d10d-2eea-4866-a833-59cb87ba67e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.288167 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.288141    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c6d10d-2eea-4866-a833-59cb87ba67e3-tls-certs" pod "16c6d10d-2eea-4866-a833-59cb87ba67e3" (UID: "16c6d10d-2eea-4866-a833-59cb87ba67e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:50.288284 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.288165    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-dshm" pod "16c6d10d-2eea-4866-a833-59cb87ba67e3" (UID: "16c6d10d-2eea-4866-a833-59cb87ba67e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:50.289019 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.288989    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c6d10d-2eea-4866-a833-59cb87ba67e3-kube-api-access-rfchk" pod "16c6d10d-2eea-4866-a833-59cb87ba67e3" (UID: "16c6d10d-2eea-4866-a833-59cb87ba67e3"). InnerVolumeSpecName "kube-api-access-rfchk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:50.387618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.387571    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfchk\" (UniqueName: \"kubernetes.io/projected/16c6d10d-2eea-4866-a833-59cb87ba67e3-kube-api-access-rfchk\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:50.387618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.387611    4282 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16c6d10d-2eea-4866-a833-59cb87ba67e3-tls-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:50.387618 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.387626    4282 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-dshm\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:50.387967 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:50.387637    4282 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16c6d10d-2eea-4866-a833-59cb87ba67e3-tmp-dir\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:51.086627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.086588    4282 generic.go:356] "Generic (PLEG): container finished" podID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerID="ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04" exitCode=0
Oct 30 22:35:51.086627 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.086617    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"
Oct 30 22:35:51.087148 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.086672    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" event={"ID":"16c6d10d-2eea-4866-a833-59cb87ba67e3","Type":"ContainerDied","Data":"ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04"}
Oct 30 22:35:51.087148 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.086709    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj" event={"ID":"16c6d10d-2eea-4866-a833-59cb87ba67e3","Type":"ContainerDied","Data":"92869d88daa0394c08e7ef0d683a0c693396f959e2153c375ff3d9b8425b2434"}
Oct 30 22:35:51.087148 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.086727    4282 scope.go:122] "RemoveContainer" containerID="ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04"
Oct 30 22:35:51.096481 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.096448    4282 scope.go:122] "RemoveContainer" containerID="ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04"
Oct 30 22:35:51.096738 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:35:51.096719    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04\": container with ID starting with ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04 not found: ID does not exist" containerID="ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04"
Oct 30 22:35:51.096811 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.096744    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04"} err="failed to get container status \"ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04\": rpc error: code = NotFound desc = could not find container \"ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04\": container with ID starting with ec4aab50fdb444a5e5e8fcedcfe1a7185b0038052ecd6e839dc75f8906fafc04 not found: ID does not exist"
Oct 30 22:35:51.108158 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.108137    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"]
Oct 30 22:35:51.112116 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.112097    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-2-w4-939a16-kserve-866f847b7-q65jj"]
Oct 30 22:35:51.462631 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:51.462599    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" path="/var/lib/kubelet/pods/16c6d10d-2eea-4866-a833-59cb87ba67e3/volumes"
Oct 30 22:35:52.576173 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:52.576137    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:35:57.973489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:35:57.973393    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-r9qsh"]
Oct 30 22:36:02.315984 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:02.315939    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"]
Oct 30 22:36:02.316542 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:02.316246    4282 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" containerID="cri-o://5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735" gracePeriod=30
Oct 30 22:36:04.150916 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:04.150883    4282 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:36:04.151333 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:04.150963    4282 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:36:04.648252 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:36:04.648219    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:36:04.696999 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:04.696972    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-r9qsh_c648ae22-6494-4e9a-a4c9-bfc6e595c947/limitador/0.log"
Oct 30 22:36:05.652720 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:05.652680    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-869599b8cc-mkw2r_4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a/manager/0.log"
Oct 30 22:36:06.062755 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:06.062725    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pnfdf_82a523cc-b6a3-4643-a0bf-aa2547f56eea/manager/0.log"
Oct 30 22:36:07.363226 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.363203    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7hk96_655934c6-c911-4bbd-bc6e-7ce2dd3b368e/proxy/0.log"
Oct 30 22:36:07.564845 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.564825    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 22:36:07.654749 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.654680    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-token\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-token\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.654749 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.654728    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-envoy\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.654994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.654762    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad0fdf8-6237-440f-b396-c1eebd36b982-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad0fdf8-6237-440f-b396-c1eebd36b982-istiod-ca-cert\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.654994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.654812    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-podinfo\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.654994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.654854    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-certs\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.654994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.654881    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-kube-api-access-7wrq2\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-kube-api-access-7wrq2\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.654994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.654910    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-socket\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.654994 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.654971    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-credential-socket\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.655356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655061    4282 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-data\") pod \"cad0fdf8-6237-440f-b396-c1eebd36b982\" (UID: \"cad0fdf8-6237-440f-b396-c1eebd36b982\") "
Oct 30 22:36:07.655356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655124    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-certs" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:07.655356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655220    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad0fdf8-6237-440f-b396-c1eebd36b982-istiod-ca-cert" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:07.655356 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655350    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-socket" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:07.655629 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655423    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-data" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:07.655629 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655591    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-credential-socket" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:07.655831 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655811    4282 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-credential-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:07.655872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655837    4282 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-data\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:07.655872 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655853    4282 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad0fdf8-6237-440f-b396-c1eebd36b982-istiod-ca-cert\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:07.655944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655869    4282 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-certs\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:07.655944 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.655883    4282 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-workload-socket\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:07.657152 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.657125    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-token" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:07.657262 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.657130    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-envoy" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:07.657753 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.657720    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-podinfo" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:07.657753 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.657722    4282 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-kube-api-access-7wrq2" pod "cad0fdf8-6237-440f-b396-c1eebd36b982" (UID: "cad0fdf8-6237-440f-b396-c1eebd36b982"). InnerVolumeSpecName "kube-api-access-7wrq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:07.756528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.756500    4282 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-token\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:07.756528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.756522    4282 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-envoy\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:07.756528 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.756532    4282 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad0fdf8-6237-440f-b396-c1eebd36b982-istio-podinfo\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:07.756753 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:07.756541    4282 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wrq2\" (UniqueName: \"kubernetes.io/projected/cad0fdf8-6237-440f-b396-c1eebd36b982-kube-api-access-7wrq2\") on node \"ip-10-0-131-245.us-east-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:08.165973 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.165937    4282 generic.go:356] "Generic (PLEG): container finished" podID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerID="5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735" exitCode=0
Oct 30 22:36:08.166149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.165988    4282 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"
Oct 30 22:36:08.166149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.166028    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" event={"ID":"cad0fdf8-6237-440f-b396-c1eebd36b982","Type":"ContainerDied","Data":"5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735"}
Oct 30 22:36:08.166149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.166087    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql" event={"ID":"cad0fdf8-6237-440f-b396-c1eebd36b982","Type":"ContainerDied","Data":"7be8bcdc362b1f32dbaee03f71902ebc12a247c1ca0583909c3c8dbd02e64b20"}
Oct 30 22:36:08.166149 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.166116    4282 scope.go:122] "RemoveContainer" containerID="5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735"
Oct 30 22:36:08.177523 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.177493    4282 scope.go:122] "RemoveContainer" containerID="5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735"
Oct 30 22:36:08.177913 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:36:08.177880    4282 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735\": container with ID starting with 5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735 not found: ID does not exist" containerID="5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735"
Oct 30 22:36:08.178027 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.177926    4282 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735"} err="failed to get container status \"5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735\": rpc error: code = NotFound desc = could not find container \"5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735\": container with ID starting with 5a5a7131c033f7e292f9b5b8bdcf760b24582664d7b6fc18e926015424b54735 not found: ID does not exist"
Oct 30 22:36:08.196160 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.196139    4282 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"]
Oct 30 22:36:08.200560 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:08.200538    4282 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-zj5ql"]
Oct 30 22:36:09.462550 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:09.462519    4282 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" path="/var/lib/kubelet/pods/cad0fdf8-6237-440f-b396-c1eebd36b982/volumes"
Oct 30 22:36:18.739420 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:18.739390    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-869599b8cc-mkw2r_4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a/manager/0.log"
Oct 30 22:36:19.125802 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:19.125726    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pnfdf_82a523cc-b6a3-4643-a0bf-aa2547f56eea/manager/0.log"
Oct 30 22:36:20.443541 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:20.443507    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7hk96_655934c6-c911-4bbd-bc6e-7ce2dd3b368e/proxy/0.log"
Oct 30 22:36:21.222806 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:21.222775    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-dffdd5df6-kgddg_445638dc-b7cd-4036-a8f8-83ed400affa8/maas-api/0.log"
Oct 30 22:36:22.887366 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:22.887332    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-da804873-5hzsj_ba418436-3047-4b2d-9efa-d0a8b9582aaa/revoke-keys/0.log"
Oct 30 22:36:25.828213 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:25.828175    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-r9qsh_c648ae22-6494-4e9a-a4c9-bfc6e595c947/limitador/0.log"
Oct 30 22:36:26.984113 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:26.984082    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2clv5_e4e8949f-3858-4a3a-8028-00fc7a736f9f/discovery/0.log"
Oct 30 22:36:27.707992 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:27.707964    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78559bccc4-wjlbh_59de6acd-8145-41f4-aa19-7f5696f7d790/kube-auth-proxy/0.log"
Oct 30 22:36:28.301893 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:28.301858    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-c5n7j_f536d905-2529-4495-80ff-1d16c5e2cc8a/istio-proxy/0.log"
Oct 30 22:36:29.071854 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:29.071814    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-vgddj_8c079c2b-d4bf-4a25-b5fb-437d434060fe/istio-proxy/0.log"
Oct 30 22:36:30.768973 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:30.768942    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-jpht5_2141267f-7dad-4ced-9a09-ffcf9d7524a6/main/0.log"
Oct 30 22:36:31.134035 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:31.133961    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-qh7rx_4ad8c14f-8ea8-4d5d-911e-9cddd2ecf6a8/main/0.log"
Oct 30 22:36:31.539157 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:31.539121    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qh4jc_76d92e07-0511-4c67-abdc-bdc12605768b/main/0.log"
Oct 30 22:36:31.904950 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:31.904846    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-fx92c_6e3b6173-4aa6-4e05-8829-0b6a47cccd38/main/0.log"
Oct 30 22:36:34.647918 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:36:34.647884    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:36:43.031874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.031838    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5v2vp/must-gather-sgbsq"]
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032190    4282 cpu_manager.go:414] "containerMap: removing container" podUID="ba418436-3047-4b2d-9efa-d0a8b9582aaa" containerName="revoke-keys"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032200    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba418436-3047-4b2d-9efa-d0a8b9582aaa" containerName="revoke-keys"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032217    4282 cpu_manager.go:414] "containerMap: removing container" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032221    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032236    4282 cpu_manager.go:414] "containerMap: removing container" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032241    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032250    4282 cpu_manager.go:414] "containerMap: removing container" podUID="fbca5f55-1e74-457f-8f10-01e4601e8566" containerName="payload-processing"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032254    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbca5f55-1e74-457f-8f10-01e4601e8566" containerName="payload-processing"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032260    4282 cpu_manager.go:414] "containerMap: removing container" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032264    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032285    4282 cpu_manager.go:414] "containerMap: removing container" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032290    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032300    4282 cpu_manager.go:414] "containerMap: removing container" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032304    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e713fa3-0235-4cb6-aea0-65fa0591293d" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032310    4282 cpu_manager.go:414] "containerMap: removing container" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy"
Oct 30 22:36:43.032305 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032315    4282 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy"
Oct 30 22:36:43.033045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032373    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c6d10d-2eea-4866-a833-59cb87ba67e3" containerName="main"
Oct 30 22:36:43.033045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032384    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad0fdf8-6237-440f-b396-c1eebd36b982" containerName="istio-proxy"
Oct 30 22:36:43.033045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032389    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbca5f55-1e74-457f-8f10-01e4601e8566" containerName="payload-processing"
Oct 30 22:36:43.033045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032395    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd34ab38-6807-4efc-961f-6b10adb0f268" containerName="istio-proxy"
Oct 30 22:36:43.033045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032400    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c6b36b-5c4f-4472-a233-9e6bbfdc1e6c" containerName="istio-proxy"
Oct 30 22:36:43.033045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032407    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d838db-a10d-4705-8c50-09e5ce26689a" containerName="istio-proxy"
Oct 30 22:36:43.033045 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.032413    4282 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba418436-3047-4b2d-9efa-d0a8b9582aaa" containerName="revoke-keys"
Oct 30 22:36:43.038115 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.038083    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v2vp/must-gather-sgbsq"
Oct 30 22:36:43.048313 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.048289    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5v2vp\"/\"default-dockercfg-dsdt9\""
Oct 30 22:36:43.048766 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.048747    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v2vp\"/\"kube-root-ca.crt\""
Oct 30 22:36:43.048880 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.048782    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5v2vp\"/\"openshift-service-ca.crt\""
Oct 30 22:36:43.055750 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.055727    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v2vp/must-gather-sgbsq"]
Oct 30 22:36:43.128222 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.128192    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k2j9d/must-gather-v6652"]
Oct 30 22:36:43.132065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.132045    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2j9d/must-gather-v6652"
Oct 30 22:36:43.133086 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.133053    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x57jw/must-gather-ccgz2"]
Oct 30 22:36:43.136425 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.136407    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x57jw/must-gather-ccgz2"
Oct 30 22:36:43.143257 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.143236    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2j9d/must-gather-v6652"]
Oct 30 22:36:43.146973 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.146955    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x57jw/must-gather-ccgz2"]
Oct 30 22:36:43.149515 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.149490    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k2j9d\"/\"default-dockercfg-xhph6\""
Oct 30 22:36:43.149625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.149558    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k2j9d\"/\"kube-root-ca.crt\""
Oct 30 22:36:43.149625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.149569    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k2j9d\"/\"openshift-service-ca.crt\""
Oct 30 22:36:43.149625 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.149613    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x57jw\"/\"openshift-service-ca.crt\""
Oct 30 22:36:43.149805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.149627    4282 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x57jw\"/\"kube-root-ca.crt\""
Oct 30 22:36:43.149805 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.149614    4282 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x57jw\"/\"default-dockercfg-rmvmr\""
Oct 30 22:36:43.163341 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.163311    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxzwx\" (UniqueName: \"kubernetes.io/projected/78d8294b-e359-499d-a9a8-ab8a94fd36fe-kube-api-access-qxzwx\") pod \"must-gather-sgbsq\" (UID: \"78d8294b-e359-499d-a9a8-ab8a94fd36fe\") " pod="openshift-must-gather-5v2vp/must-gather-sgbsq"
Oct 30 22:36:43.163461 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.163431    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/78d8294b-e359-499d-a9a8-ab8a94fd36fe-must-gather-output\") pod \"must-gather-sgbsq\" (UID: \"78d8294b-e359-499d-a9a8-ab8a94fd36fe\") " pod="openshift-must-gather-5v2vp/must-gather-sgbsq"
Oct 30 22:36:43.264375 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.264345    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxzwx\" (UniqueName: \"kubernetes.io/projected/78d8294b-e359-499d-a9a8-ab8a94fd36fe-kube-api-access-qxzwx\") pod \"must-gather-sgbsq\" (UID: \"78d8294b-e359-499d-a9a8-ab8a94fd36fe\") " pod="openshift-must-gather-5v2vp/must-gather-sgbsq"
Oct 30 22:36:43.264588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.264396    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2r4q\" (UniqueName: \"kubernetes.io/projected/851abc1a-8aee-4c0e-9cf4-006238d5d6d2-kube-api-access-n2r4q\") pod \"must-gather-ccgz2\" (UID: \"851abc1a-8aee-4c0e-9cf4-006238d5d6d2\") " pod="openshift-must-gather-x57jw/must-gather-ccgz2"
Oct 30 22:36:43.264588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.264421    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/088274e2-022c-402d-bfbe-4d4add61b17c-must-gather-output\") pod \"must-gather-v6652\" (UID: \"088274e2-022c-402d-bfbe-4d4add61b17c\") " pod="openshift-must-gather-k2j9d/must-gather-v6652"
Oct 30 22:36:43.264588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.264513    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpd9\" (UniqueName: \"kubernetes.io/projected/088274e2-022c-402d-bfbe-4d4add61b17c-kube-api-access-lcpd9\") pod \"must-gather-v6652\" (UID: \"088274e2-022c-402d-bfbe-4d4add61b17c\") " pod="openshift-must-gather-k2j9d/must-gather-v6652"
Oct 30 22:36:43.264588 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.264573    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/851abc1a-8aee-4c0e-9cf4-006238d5d6d2-must-gather-output\") pod \"must-gather-ccgz2\" (UID: \"851abc1a-8aee-4c0e-9cf4-006238d5d6d2\") " pod="openshift-must-gather-x57jw/must-gather-ccgz2"
Oct 30 22:36:43.264780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.264657    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/78d8294b-e359-499d-a9a8-ab8a94fd36fe-must-gather-output\") pod \"must-gather-sgbsq\" (UID: \"78d8294b-e359-499d-a9a8-ab8a94fd36fe\") " pod="openshift-must-gather-5v2vp/must-gather-sgbsq"
Oct 30 22:36:43.264941 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.264926    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/78d8294b-e359-499d-a9a8-ab8a94fd36fe-must-gather-output\") pod \"must-gather-sgbsq\" (UID: \"78d8294b-e359-499d-a9a8-ab8a94fd36fe\") " pod="openshift-must-gather-5v2vp/must-gather-sgbsq"
Oct 30 22:36:43.275834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.275814    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxzwx\" (UniqueName: \"kubernetes.io/projected/78d8294b-e359-499d-a9a8-ab8a94fd36fe-kube-api-access-qxzwx\") pod \"must-gather-sgbsq\" (UID: \"78d8294b-e359-499d-a9a8-ab8a94fd36fe\") " pod="openshift-must-gather-5v2vp/must-gather-sgbsq"
Oct 30 22:36:43.365261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.365194    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2r4q\" (UniqueName: \"kubernetes.io/projected/851abc1a-8aee-4c0e-9cf4-006238d5d6d2-kube-api-access-n2r4q\") pod \"must-gather-ccgz2\" (UID: \"851abc1a-8aee-4c0e-9cf4-006238d5d6d2\") " pod="openshift-must-gather-x57jw/must-gather-ccgz2"
Oct 30 22:36:43.365261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.365232    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/088274e2-022c-402d-bfbe-4d4add61b17c-must-gather-output\") pod \"must-gather-v6652\" (UID: \"088274e2-022c-402d-bfbe-4d4add61b17c\") " pod="openshift-must-gather-k2j9d/must-gather-v6652"
Oct 30 22:36:43.365261 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.365254    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpd9\" (UniqueName: \"kubernetes.io/projected/088274e2-022c-402d-bfbe-4d4add61b17c-kube-api-access-lcpd9\") pod \"must-gather-v6652\" (UID: \"088274e2-022c-402d-bfbe-4d4add61b17c\") " pod="openshift-must-gather-k2j9d/must-gather-v6652"
Oct 30 22:36:43.365566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.365304    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/851abc1a-8aee-4c0e-9cf4-006238d5d6d2-must-gather-output\") pod \"must-gather-ccgz2\" (UID: \"851abc1a-8aee-4c0e-9cf4-006238d5d6d2\") " pod="openshift-must-gather-x57jw/must-gather-ccgz2"
Oct 30 22:36:43.365566 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.365548    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/088274e2-022c-402d-bfbe-4d4add61b17c-must-gather-output\") pod \"must-gather-v6652\" (UID: \"088274e2-022c-402d-bfbe-4d4add61b17c\") " pod="openshift-must-gather-k2j9d/must-gather-v6652"
Oct 30 22:36:43.365689 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.365656    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/851abc1a-8aee-4c0e-9cf4-006238d5d6d2-must-gather-output\") pod \"must-gather-ccgz2\" (UID: \"851abc1a-8aee-4c0e-9cf4-006238d5d6d2\") " pod="openshift-must-gather-x57jw/must-gather-ccgz2"
Oct 30 22:36:43.369722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.369708    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5v2vp/must-gather-sgbsq"
Oct 30 22:36:43.377003 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.376976    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpd9\" (UniqueName: \"kubernetes.io/projected/088274e2-022c-402d-bfbe-4d4add61b17c-kube-api-access-lcpd9\") pod \"must-gather-v6652\" (UID: \"088274e2-022c-402d-bfbe-4d4add61b17c\") " pod="openshift-must-gather-k2j9d/must-gather-v6652"
Oct 30 22:36:43.377097 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.376980    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2r4q\" (UniqueName: \"kubernetes.io/projected/851abc1a-8aee-4c0e-9cf4-006238d5d6d2-kube-api-access-n2r4q\") pod \"must-gather-ccgz2\" (UID: \"851abc1a-8aee-4c0e-9cf4-006238d5d6d2\") " pod="openshift-must-gather-x57jw/must-gather-ccgz2"
Oct 30 22:36:43.447135 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.447103    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2j9d/must-gather-v6652"
Oct 30 22:36:43.462230 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.462208    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x57jw/must-gather-ccgz2"
Oct 30 22:36:43.644774 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.644697    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5v2vp/must-gather-sgbsq"]
Oct 30 22:36:43.648092 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:36:43.648066    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78d8294b_e359_499d_a9a8_ab8a94fd36fe.slice/crio-a6a7fd85ebe31506635f6cb01d4ae230fafd353b381854c27dbffce21876f7ab WatchSource:0}: Error finding container a6a7fd85ebe31506635f6cb01d4ae230fafd353b381854c27dbffce21876f7ab: Status 404 returned error can't find the container with id a6a7fd85ebe31506635f6cb01d4ae230fafd353b381854c27dbffce21876f7ab
Oct 30 22:36:43.649905 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.649884    4282 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:36:43.729780 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.729757    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2j9d/must-gather-v6652"]
Oct 30 22:36:43.731728 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:36:43.731703    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod088274e2_022c_402d_bfbe_4d4add61b17c.slice/crio-b0860994fb93b36abfdd8c2d5f50ac8bbb2f248cd0be2edbe6f1b6fd3435eae6 WatchSource:0}: Error finding container b0860994fb93b36abfdd8c2d5f50ac8bbb2f248cd0be2edbe6f1b6fd3435eae6: Status 404 returned error can't find the container with id b0860994fb93b36abfdd8c2d5f50ac8bbb2f248cd0be2edbe6f1b6fd3435eae6
Oct 30 22:36:43.747345 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:43.747319    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x57jw/must-gather-ccgz2"]
Oct 30 22:36:43.750682 ip-10-0-131-245 kubenswrapper[4282]: W1030 22:36:43.750658    4282 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod851abc1a_8aee_4c0e_9cf4_006238d5d6d2.slice/crio-41d9aa22fc76d768c2fe8a252c065b3130ebace125cec38198a945e468cae545 WatchSource:0}: Error finding container 41d9aa22fc76d768c2fe8a252c065b3130ebace125cec38198a945e468cae545: Status 404 returned error can't find the container with id 41d9aa22fc76d768c2fe8a252c065b3130ebace125cec38198a945e468cae545
Oct 30 22:36:44.333874 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:44.333842    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v2vp/must-gather-sgbsq" event={"ID":"78d8294b-e359-499d-a9a8-ab8a94fd36fe","Type":"ContainerStarted","Data":"a6a7fd85ebe31506635f6cb01d4ae230fafd353b381854c27dbffce21876f7ab"}
Oct 30 22:36:44.334991 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:44.334943    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x57jw/must-gather-ccgz2" event={"ID":"851abc1a-8aee-4c0e-9cf4-006238d5d6d2","Type":"ContainerStarted","Data":"41d9aa22fc76d768c2fe8a252c065b3130ebace125cec38198a945e468cae545"}
Oct 30 22:36:44.340211 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:44.340186    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/must-gather-v6652" event={"ID":"088274e2-022c-402d-bfbe-4d4add61b17c","Type":"ContainerStarted","Data":"b0860994fb93b36abfdd8c2d5f50ac8bbb2f248cd0be2edbe6f1b6fd3435eae6"}
Oct 30 22:36:45.364612 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:45.364573    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/must-gather-v6652" event={"ID":"088274e2-022c-402d-bfbe-4d4add61b17c","Type":"ContainerStarted","Data":"e4e5789f45c80acc031c8fb41d286193f2c9dcdce178447316029605ca0e352e"}
Oct 30 22:36:45.365126 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:45.364631    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/must-gather-v6652" event={"ID":"088274e2-022c-402d-bfbe-4d4add61b17c","Type":"ContainerStarted","Data":"a9f87b4be259ca1a5c1c0ab36b6e447343eb2f5724398f8b8d4c94c1f93f2652"}
Oct 30 22:36:45.383815 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:45.383741    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k2j9d/must-gather-v6652" podStartSLOduration=1.494180811 podStartE2EDuration="2.383722657s" totalImagesPullingTime="889.541846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:45.380687996 +0000 UTC m=+3424.038784233" watchObservedRunningTime="2026-10-30 22:36:45.383722657 +0000 UTC m=+3424.041818882"
Oct 30 22:36:47.967241 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:47.967114    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r5xwr_aa1dbadd-b4a7-44cb-b35a-768e9315acfc/global-pull-secret-syncer/0.log"
Oct 30 22:36:48.376244 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:48.376167    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sqkkh_cc4b5658-0527-4a3c-aa15-5f8d74b8d8a9/konnectivity-agent/0.log"
Oct 30 22:36:48.469228 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:48.466653    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-245.us-east-2.compute.internal_a1e9e5008f7a1caba4d99ed76ccdf2ff/haproxy/0.log"
Oct 30 22:36:56.669376 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:56.669288    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-r9qsh_c648ae22-6494-4e9a-a4c9-bfc6e595c947/limitador/0.log"
Oct 30 22:36:58.237533 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.237432    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dab523b5-a6c7-4ae1-811d-15509ccf868b/alertmanager/0.log"
Oct 30 22:36:58.277626 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.275131    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dab523b5-a6c7-4ae1-811d-15509ccf868b/config-reloader/0.log"
Oct 30 22:36:58.301443 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.301368    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dab523b5-a6c7-4ae1-811d-15509ccf868b/kube-rbac-proxy-web/0.log"
Oct 30 22:36:58.336899 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.336442    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dab523b5-a6c7-4ae1-811d-15509ccf868b/kube-rbac-proxy/0.log"
Oct 30 22:36:58.366456 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.366403    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dab523b5-a6c7-4ae1-811d-15509ccf868b/kube-rbac-proxy-metric/0.log"
Oct 30 22:36:58.397563 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.397511    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dab523b5-a6c7-4ae1-811d-15509ccf868b/prom-label-proxy/0.log"
Oct 30 22:36:58.430969 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.430936    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_dab523b5-a6c7-4ae1-811d-15509ccf868b/init-config-reloader/0.log"
Oct 30 22:36:58.809949 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.809889    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jcp77_10f05285-32c1-4416-82fd-dd2d81a3a8bb/node-exporter/0.log"
Oct 30 22:36:58.855430 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.855339    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jcp77_10f05285-32c1-4416-82fd-dd2d81a3a8bb/kube-rbac-proxy/0.log"
Oct 30 22:36:58.888260 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:58.888082    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jcp77_10f05285-32c1-4416-82fd-dd2d81a3a8bb/init-textfile/0.log"
Oct 30 22:36:59.458358 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:59.458326    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-t4fkx_385e7e99-d468-4345-b432-dd1565ef5c5c/prometheus-operator-admission-webhook/0.log"
Oct 30 22:36:59.496926 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:59.496895    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-546f884c57-bf88n_551f22c1-58c2-440d-9326-d78f3a23941d/telemeter-client/0.log"
Oct 30 22:36:59.526069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:59.526035    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-546f884c57-bf88n_551f22c1-58c2-440d-9326-d78f3a23941d/reload/0.log"
Oct 30 22:36:59.567085 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:36:59.567021    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-546f884c57-bf88n_551f22c1-58c2-440d-9326-d78f3a23941d/kube-rbac-proxy/0.log"
Oct 30 22:37:00.212757 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.211697    4282 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"]
Oct 30 22:37:00.362065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.361257    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"]
Oct 30 22:37:00.362065 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.361393    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.423279 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.423146    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-lib-modules\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.423753 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.423249    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fj8\" (UniqueName: \"kubernetes.io/projected/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-kube-api-access-l2fj8\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.424419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.424173    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-podres\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.424419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.424222    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-proc\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.424419 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.424282    4282 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-sys\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.530877    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-podres\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.530942    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-proc\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.530989    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-sys\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.531014    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-lib-modules\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.531077    4282 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fj8\" (UniqueName: \"kubernetes.io/projected/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-kube-api-access-l2fj8\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.531334    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-proc\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.531420    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-podres\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.531493    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-sys\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.532495 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.531546    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-lib-modules\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.553496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.549552    4282 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fj8\" (UniqueName: \"kubernetes.io/projected/1f0e1b66-7c3a-44e0-95bd-bcf5158d576d-kube-api-access-l2fj8\") pod \"perf-node-gather-daemonset-v7lz4\" (UID: \"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:00.742283 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:00.741655    4282 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:01.048484 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:01.048426    4282 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"]
Oct 30 22:37:01.615478 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:01.615402    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v2vp/must-gather-sgbsq" event={"ID":"78d8294b-e359-499d-a9a8-ab8a94fd36fe","Type":"ContainerStarted","Data":"7cfe9c88ddf8f46ab89a707f12ee7bcb83e32ddd852d85c5d381553591274e1e"}
Oct 30 22:37:01.618520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:01.617755    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4" event={"ID":"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d","Type":"ContainerStarted","Data":"7ace2c8893c75906ddf825262582e6ed50b878401bea55fba3dffc3d980517d7"}
Oct 30 22:37:01.618520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:01.617800    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4" event={"ID":"1f0e1b66-7c3a-44e0-95bd-bcf5158d576d","Type":"ContainerStarted","Data":"a0e91ee3269d438bd1388f5af7356acc24cf682a3629f9e89fc4ab8a57345b4b"}
Oct 30 22:37:01.618520 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:01.618442    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:01.644322 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:01.642684    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4" podStartSLOduration=1.6426653500000001 podStartE2EDuration="1.64266535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:01.642066183 +0000 UTC m=+3440.300162407" watchObservedRunningTime="2026-10-30 22:37:01.64266535 +0000 UTC m=+3440.300761577"
Oct 30 22:37:02.634667 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:02.633420    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5v2vp/must-gather-sgbsq" event={"ID":"78d8294b-e359-499d-a9a8-ab8a94fd36fe","Type":"ContainerStarted","Data":"400229c9d7e710304bda678ac8bb1da13531abaa7e2209ba22d70712a11404a9"}
Oct 30 22:37:02.637839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:02.637751    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x57jw/must-gather-ccgz2" event={"ID":"851abc1a-8aee-4c0e-9cf4-006238d5d6d2","Type":"ContainerStarted","Data":"8ce2dbd6f1e8a1b2130fecbd297304fa2bb3c909bfaee26f5ba1b7d7ad789a8a"}
Oct 30 22:37:02.637839 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:02.637807    4282 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x57jw/must-gather-ccgz2" event={"ID":"851abc1a-8aee-4c0e-9cf4-006238d5d6d2","Type":"ContainerStarted","Data":"b8d5bb0d58b85c5bf32556ebc8e75df47bfd298901a13d1d5c700d96596cdbf3"}
Oct 30 22:37:02.655834 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:02.655776    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5v2vp/must-gather-sgbsq" podStartSLOduration=2.0306547 podStartE2EDuration="19.655758482s" totalImagesPullingTime="17.625103782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:02.652840779 +0000 UTC m=+3441.310937004" watchObservedRunningTime="2026-10-30 22:37:02.655758482 +0000 UTC m=+3441.313854706"
Oct 30 22:37:02.672885 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:02.672831    4282 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-x57jw/must-gather-ccgz2" podStartSLOduration=1.97147348 podStartE2EDuration="19.672817099s" totalImagesPullingTime="17.701343619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:02.668358059 +0000 UTC m=+3441.326454279" watchObservedRunningTime="2026-10-30 22:37:02.672817099 +0000 UTC m=+3441.330913319"
Oct 30 22:37:03.179737 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:03.179111    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-cbc7cf9f7-vvh2w_f3baa9af-6804-4f85-a6e4-0661ebfd541f/console/0.log"
Oct 30 22:37:03.235069 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:03.235042    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-vp2lj_51815808-5f76-4970-a434-88c3e11b2348/download-server/0.log"
Oct 30 22:37:03.548207 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:03.548133    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-869599b8cc-mkw2r_4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a/manager/0.log"
Oct 30 22:37:03.689634 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:03.689601    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-869599b8cc-mkw2r_4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a/manager/0.log"
Oct 30 22:37:03.709715 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:03.709687    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pnfdf_82a523cc-b6a3-4643-a0bf-aa2547f56eea/manager/0.log"
Oct 30 22:37:03.779531 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:03.779496    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pnfdf_82a523cc-b6a3-4643-a0bf-aa2547f56eea/manager/0.log"
Oct 30 22:37:04.678852 ip-10-0-131-245 kubenswrapper[4282]: E1030 22:37:04.678626    4282 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-9l44m" containerName="extractor"
Oct 30 22:37:06.033717 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:06.033688    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-764zc_02da34f8-540c-4b70-a9d3-6cf004c05a10/dns/0.log"
Oct 30 22:37:06.067276 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:06.067247    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-764zc_02da34f8-540c-4b70-a9d3-6cf004c05a10/kube-rbac-proxy/0.log"
Oct 30 22:37:06.259929 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:06.259902    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-82lq5_4989e2a1-ce7d-4da1-b626-946742bf4897/dns-node-resolver/0.log"
Oct 30 22:37:06.531975 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:06.531946    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7hk96_655934c6-c911-4bbd-bc6e-7ce2dd3b368e/proxy/0.log"
Oct 30 22:37:06.598301 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:06.598259    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7hk96_655934c6-c911-4bbd-bc6e-7ce2dd3b368e/proxy/0.log"
Oct 30 22:37:07.299795 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:07.299760    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6886d97769-h7tkk_ed9aa114-485b-4f90-b18e-f3bfb0493287/registry/0.log"
Oct 30 22:37:07.335907 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:07.334881    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kbfjc_12528140-5996-44d5-b1ea-a1eea5b4a65c/node-ca/0.log"
Oct 30 22:37:07.870488 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:07.870381    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-869599b8cc-mkw2r_4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a/manager/0.log"
Oct 30 22:37:08.010934 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:08.010875    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pnfdf_82a523cc-b6a3-4643-a0bf-aa2547f56eea/manager/0.log"
Oct 30 22:37:08.015985 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:08.015958    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-869599b8cc-mkw2r_4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a/manager/0.log"
Oct 30 22:37:08.150728 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:08.150641    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pnfdf_82a523cc-b6a3-4643-a0bf-aa2547f56eea/manager/0.log"
Oct 30 22:37:08.669493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:08.669214    4282 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-v7lz4"
Oct 30 22:37:10.910494 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:10.908218    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7hk96_655934c6-c911-4bbd-bc6e-7ce2dd3b368e/proxy/0.log"
Oct 30 22:37:11.022436 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:11.022404    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7hk96_655934c6-c911-4bbd-bc6e-7ce2dd3b368e/proxy/0.log"
Oct 30 22:37:12.946623 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:12.946593    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-869599b8cc-mkw2r_4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a/manager/0.log"
Oct 30 22:37:13.015496 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:13.015437    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-869599b8cc-mkw2r_4ba1f36f-e586-4ba1-a3f5-7dfad8fd363a/manager/0.log"
Oct 30 22:37:13.168493 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:13.168445    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pnfdf_82a523cc-b6a3-4643-a0bf-aa2547f56eea/manager/0.log"
Oct 30 22:37:13.220620 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:13.220545    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pnfdf_82a523cc-b6a3-4643-a0bf-aa2547f56eea/manager/0.log"
Oct 30 22:37:15.321722 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:15.321688    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-2clv5_e4e8949f-3858-4a3a-8028-00fc7a736f9f/discovery/0.log"
Oct 30 22:37:16.091489 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:16.090005    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7hk96_655934c6-c911-4bbd-bc6e-7ce2dd3b368e/proxy/0.log"
Oct 30 22:37:16.140491 ip-10-0-131-245 kubenswrapper[4282]: I1030 22:37:16.140138    4282 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-7hk96_655934c6-c911-4bbd-bc6e-7ce2dd3b368e/proxy/0.log"